moeburn Posted July 8, 2013 Share Posted July 8, 2013 I'm using my Android smartphone to send accelerometer sensor data wirelessly to my PC via UDP, and I have a Python script to display the data on screen: import socket, traceback host = '' port = 5555 s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1) s.bind((host, port)) while 1: try: message, address = s.recvfrom(8192) print (message) except (KeyboardInterrupt, SystemExit): raise except: traceback.print_exc() And it works fine, but I don't really know anything about Python. I want to write the data to a CSV file instead of the screen, how would I do that? Link to comment Share on other sites More sharing options...
0 Lant Posted July 8, 2013 Share Posted July 8, 2013 Adapted from http://docs.python.org/2/library/csv.html#csv.writer import csv with open('eggs.csv', 'wb') as csvfile: spamwriter = csv.writer(csvfile) spamwriter.writerow([message]) Also if you are using python 2.x you might want to add this to the top of your script (see: http://docs.python.org/2/library/__future__.html) from __future__ import print_function, with_statement Link to comment Share on other sites More sharing options...
0 moeburn Posted July 8, 2013 Author Share Posted July 8, 2013 I'm using Python 3.3.2, I just installed it. I haven't really programmed anything since the days of Visual Basic 6. So I wasn't sure where to add that code you mentioned, so I added it like such: import socket, traceback import csv host = '' port = 5555 s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1) s.bind((host, port)) while 1: try: message, address = s.recvfrom(8192) print (message) with open('eggs.csv', 'wb') as csvfile: spamwriter = csv.writer(csvfile) spamwriter.writerow([message]) except (KeyboardInterrupt, SystemExit): raise except: traceback.print_exc() And I get these errors: Link to comment Share on other sites More sharing options...
0 Karl L. Posted July 8, 2013 Share Posted July 8, 2013 You have the right idea, but it's probably not good to open and close the CSV file so many times. The following script runs with no exceptions on my Debian 7.1 box with Python 3.2. I assumed the type of data you would be sending and tested it with netcat, so it should work. #!/usr/bin/env python3 import socket import traceback import csv host = '' port = 5555 csvf = 'accelerometer.csv' s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1) s.bind((host, port)) with open(csvf, 'w', newline='') as csv_handle: csv_writer = csv.writer(csv_handle, delimiter=',') while 1: try: message, address = s.recvfrom(8192) print(message) csv_writer.writerow(message) except(KeyboardInterrupt, SystemExit): raise except: traceback.print_exc() Link to comment Share on other sites More sharing options...
0 moeburn Posted July 9, 2013 Author Share Posted July 9, 2013 Well, at least the CSV was written without errors. Now it's just writing in the wrong format. I've attached a screenshot with the proper data display format on the top left, the code on the top right, and the CSV result on the bottom. I think those numbers are the ASCII codes for the actual text? I tried adding , encoding='utf8' after newline='' but that didn't fix it. Link to comment Share on other sites More sharing options...
0 Karl L. Posted July 9, 2013 Share Posted July 9, 2013 The CSV module was writing the integer value of each byte of the string to the file. Unlike Python 2, Python 3 differentiates between a text string and byte string. The message being returned from the socket is a byte string. I probably should have checked the sanity of my output file last time. Try the following script instead: #!/usr/bin/env python3 import socket import traceback import csv import re host = '' port = 5555 csvf = 'accelerometer.csv' s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1) s.bind((host, port)) with open(csvf, 'a', newline='') as csv_handle: csv_writer = csv.writer(csv_handle, delimiter=',') while 1: try: message, address = s.recvfrom(8192) print(message) message = str(message) if message[0] == 'b': message = message[2:-1] data = [] for x in iter(message.split(', ')): if re.search('^[-]{0,1}[0-9]+\.[0-9]+$', x) != None: data.append(float(x)) elif re.search('^[-]{0,1}[0-9]$', x) != None: data.append(int(x)) elif re.search('^[-]{0,1}[0-9]+\.[0-9]+,[-]{0,1}[0-9]+\.[0-9]+$', x) != None: real, img = x.split(',') real = float(real) img = float(img) data.append(complex(real, img)) else: print('unknown', x) csv_writer.writerow(data) csv_handle.flush() except(KeyboardInterrupt, SystemExit): raise except: traceback.print_exc() If anyone else would like to test it or attempt further modifications, I recorded the values in moeburn's last screenshot in a text file and wrote a simple script to read the file and send its contents to the accelerometer script. Test data file: 7406.60063, 3, 0.765, 1.334, 9.601 7406.70039, 3, 0.765, 1.374, 9.601 7406.77045, 3, 0.765, 1.334, 9.601, 5, 28.000, 8.125,-34.938 7406.80038, 3, 0.765, 1.294, 9.601 7406.11339, 3, 0.804, 1.294, 9.601 7407.17048, 3, 0.745, 1.334, 9.601 7407.31351, 3, 0.785, 1.324, 9.601, 5, 29.062, 8.000,-34.562 7407.37044, 3, 0.785, 1.383, 9.601, 5, 28.312, 7.750,-34.438 7407.41334, 3, 0.785, 1.324, 9.601 7407.44040, 3, 0.824, 1.324, 9.601 7407.53046, 3, 0.824, 1.363, 9.601, 5, 28.125, 7.375,-34.062 7407.60038, 3, 0.824, 1.363, 8.630 7407.61541, 3, 0.765, 1.520, 9.571, 5, 28.562, 7.250,-34.938 7407.62152, 3, 0.765, 1.471, 9.512 7407.63199, 3, 0.765, 1.304, 9.512 7407.65909, 3, 0.765, 1.373, 9.718 7407.66674, 3, 0.765, 1.373, 9.758 7407.68064, 3, 0.706, 1.373, 9.915 7407.70041, 3, 0.706, 1.374, 9.856 7407.72062, 3, 0.706, 1.324, 9.924 7407.73449, 3, 0.706, 1.324, 9.689 7407.74213, 3, 0.902, 1.265, 8.120 7407.75961, 3, 0.853, 1.432, 9.022, 28.750, 7.188,-34.125 Test script: #!/usr/bin/env python3 import socket host = '' port = 5555 test = 'data.txt' s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1) s.connect((host, port)) with open(test, 'r') as test_handle: for line in test_handle: line = line.rstrip() s.send(line.encode()) Link to comment Share on other sites More sharing options...
0 moeburn Posted July 9, 2013 Author Share Posted July 9, 2013 Thanks for doing all that for me! Sorry, I should have posted the sample data as text not a screenshot, but I don't know how to copy text from the Python output screen in Windows. Your latest example does a lot of processing to the data, even the data on screen is now coming up different. But there's a lot of values missing from the CSV, here's a snipped form the CSV: 15396.84775,3 15396.85702,3 15396.86359,3 15396.87,3 15396.87998,3 15396.89283,3 15396.90079,3 15396.91003,3 15396.92027,3 15396.93003,3 15396.94188,3 15396.95006,3 15396.9702,3,5 15396.98038,3 15396.9963,3 As for the data being sent, here's a breakdown: timestamp, accelSensorID, accelX, accelY, accelZ 7407.60038, 3, 0.824, 1.363, 8.630 occasionally magnet sensor also sends data when magnet sensor input changes: timestamp, accelSensorID, accelX, accelY, accelZ, magnetSensorID, magnetX, magnetY, magnetZ 7407.61541, 3, 0.765, 1.520, 9.571, 5, 28.562, 7.250,-34.938 Link to comment Share on other sites More sharing options...
0 moeburn Posted July 9, 2013 Author Share Posted July 9, 2013 Ok I got a much simpler code that writes to the CSV like this: 1,5,8,4,9,.,8,1,1,6,3,",", ,3,",", , , ,0,.,0,0,0,",", , ,5,.,0,9,9,",", , ,8,.,9,2,4 1,5,8,4,9,.,8,2,9,8,8,",", ,3,",", , , ,0,.,0,0,0,",", , ,5,.,0,5,0,",", , ,8,.,8,6,5 1,5,8,4,9,.,8,4,9,7,5,",", ,3,",", , , ,0,.,0,4,9,",", , ,4,.,7,5,6,",", , ,8,.,6,4,9,",", ,5,",", , ,2,2,.,6,8,8,",",-,1,1,.,6,2,5,",",-,4,5,.,1,2,5 1,5,8,4,9,.,8,6,2,8,9,",", ,3,",", , , ,0,.,0,4,9,",", , ,4,.,6,8,8,",", , ,8,.,6,4,9 1,5,8,4,9,.,8,7,2,0,2,",", ,3,",", , , ,0,.,0,4,9,",", , ,4,.,6,8,8,",", , ,8,.,7,0,8 1,5,8,4,9,.,8,8,2,4,0,",", ,3,",", , , ,0,.,0,4,9,",", , ,4,.,6,8,8,",", , ,8,.,7,7,7 1,5,8,4,9,.,8,9,3,5,3,",", ,3,",", , , ,0,.,0,4,9,",", , ,4,.,7,4,6,",", , ,8,.,8,2,6 1,5,8,4,9,.,9,1,3,4,8,",", ,3,",", , , ,0,.,0,4,9,",", , ,4,.,7,0,7,",", , ,8,.,8,2,6 But I don't want all those comma delimiters in between each byte, because the android app actually adds the commas itself. Here's the code: #include libraries n stuff import socket import traceback import csv #assign variables n stuff host = '' port = 5555 csvf = 'accelerometer.csv' #do UDP stuff s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1) s.bind((host, port)) #do CSV stuff with open(csvf, 'w', newline='', encoding='ascii') as csv_handle: csv_writer = csv.writer(csv_handle) while 1: try: message, address = s.recvfrom(8192) #get data from UDP print(message) #display data on screen csv_writer.writerow(message.decode(encoding='ascii')) #write data to CSV except(KeyboardInterrupt, SystemExit): #close on exit raise except: traceback.print_exc() #display errors on screen all I did was change csv_writer.writerow(message) to csv_writer.writerow(message.decode(encoding='ascii')), and removed the delimiter=',' , although that didn't actually remove the delimiter for some reason. astropheed 1 Share Link to comment Share on other sites More sharing options...
0 astropheed Veteran Posted July 9, 2013 Veteran Share Posted July 9, 2013 I was wondering why all the RegEx when a simple x.decode() would suffice. Can you do me a favor and change that 'while 1:' to 'while True:', it irks me lol. Link to comment Share on other sites More sharing options...
0 moeburn Posted July 9, 2013 Author Share Posted July 9, 2013 Ok I got it working now! Apparently all I needed were a couple of [] brackets around the message.decode. Here's the final code: #include libraries n stuff import socket import traceback import csv #assign variables n stuff host = '' port = 5555 csvf = 'accelerometer.csv' #do UDP stuff s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1) s.bind((host, port)) #do CSV stuff with open(csvf, 'w', newline='', encoding='ascii') as csv_handle: csv_writer = csv.writer(csv_handle, delimiter=',') while 1: try: message, address = s.recvfrom(8192) #get data from UDP print(message) #display data on screen csv_writer.writerow([message.decode(encoding='ascii')]) #write data to CSV except(KeyboardInterrupt, SystemExit): #close on exit raise except: traceback.print_exc() #display errors on screen Link to comment Share on other sites More sharing options...
0 moeburn Posted July 9, 2013 Author Share Posted July 9, 2013 If anyone is interested, here is an example of some data I recorded using this script: Because the accelerometer is so sensitive, I intend to use it to measure my heart rate and breathing patterns (already confirmed possible) by taping the phone to my chest while I sleep. That's why I needed the live stream of data to my PC; in-phone sensor recording apps weren't reliable enough after a few minutes of recording. Link to comment Share on other sites More sharing options...
0 moeburn Posted July 10, 2013 Author Share Posted July 10, 2013 Hey would anyone feel like helping me improve it a bit more? At the moment, it only writes to the file once the program is closed. I'd like it to be a little more reliable than that, in case of a power failure or a Windows crash. I only know how to code it to write every time it receives a data sample, but that's 60 times a second, that's probably too often. Would anyone like to write me a piece of code to only write to the CSV once every 100kB or so? Link to comment Share on other sites More sharing options...
0 Eric Veteran Posted July 10, 2013 Veteran Share Posted July 10, 2013 If it doesn't have to be CSV you could always try using pickle/cPickle, but you have to be careful with it as it can be a security risk if your application tries to unpickle data that has been intentionally malformed. cPickle is quite fast and is the default over pickle in Python 3+. Karl L. 1 Share Link to comment Share on other sites More sharing options...
0 Karl L. Posted July 10, 2013 Share Posted July 10, 2013 Hey would anyone feel like helping me improve it a bit more? At the moment, it only writes to the file once the program is closed. I'd like it to be a little more reliable than that, in case of a power failure or a Windows crash. I only know how to code it to write every time it receives a data sample, but that's 60 times a second, that's probably too often. Would anyone like to write me a piece of code to only write to the CSV once every 100kB or so? According to the documentation for the Python open() function, you can use the buffering argument to control the size of the buffer, and thus how often data is flushed to the CSV file on disk. #!/usr/bin/env python3 import socket import traceback import csv host = '' port = 5555 csvf = 'accelerometer.csv' buff = 100 # Flush the accelerometer data to disk once every buff KB s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1) s.bind((host, port)) with open(csvf, 'w', buff * 1024, encoding='ascii', newline='') as csv_handle: csv_writer = csv.writer(csv_handle, delimiter=',') while True: try: message, address = s.recvfrom(8192) print(message) csv_writer.writerow([message.decode(encoding='ascii')]) except(KeyboardInterrupt, SystemExit): raise except: traceback.print_exc() Link to comment Share on other sites More sharing options...
Question
moeburn
I'm using my Android smartphone to send accelerometer sensor data wirelessly to my PC via UDP, and I have a Python script to display the data on screen:
And it works fine, but I don't really know anything about Python. I want to write the data to a CSV file instead of the screen, how would I do that?
Link to comment
Share on other sites
13 answers to this question
Recommended Posts