Python pandas time series interpolation and regularization -

-

i using python pandas first time. have 5-min lag traffic data in csv format:

... 2015-01-04 08:29:05,271238 2015-01-04 08:34:05,329285 2015-01-04 08:39:05,-1 2015-01-04 08:44:05,260260 2015-01-04 08:49:05,263711 ...

there several issues:

  • for timestamps there's missing data (-1)
  • missing entries (also 2/3 consecutive hours)
  • the frequency of observations not 5 minutes, loses seconds once in while

i obtain regular time series, entries every (exactly) 5 minutes (and no missing valus). have interpolated time series following code approximate -1 values code:

ts = pd.timeseries(values, index=timestamps) ts.interpolate(method='cubic', downcast='infer')

how can both interpolate , regularize frequency of observations? thank help.

change -1s nans:

ts[ts==-1] = np.nan

then resample data have 5 minute frequency. 

ts = ts.resample('5t')

note that, default, if 2 measurements fall within same 5 minute period, resample averages values together.

finally, linearly interpolate time series according time:

ts = ts.interpolate(method='time')

since looks data has 5-minute frequency, might need resample @ shorter frequency cubic or spline interpolation can smooth out curve:

import numpy np import pandas pd import matplotlib.pyplot plt  values = [271238, 329285, -1, 260260, 263711] timestamps = pd.to_datetime(['2015-01-04 08:29:05',                              '2015-01-04 08:34:05',                              '2015-01-04 08:39:05',                              '2015-01-04 08:44:05',                              '2015-01-04 08:49:05'])  ts = pd.series(values, index=timestamps) ts[ts==-1] = np.nan ts = ts.resample('t', how='mean')  ts.interpolate(method='spline', order=3).plot() ts.interpolate(method='time').plot() lines, labels = plt.gca().get_legend_handles_labels() labels = ['spline', 'time'] plt.legend(lines, labels, loc='best') plt.show()

enter image description here


Comments

Post a Comment

Popular posts from this blog

Java 3D LWJGL collision -

-
i making 3d java game lwjgl library, , wondering how add collision detection, player not go through models. i using obj models. here objloader class, loads models: package renderengine; import java.io.bufferedreader; import java.io.file; import java.io.filenotfoundexception; import java.io.filereader; import java.util.arraylist; import java.util.list; import models.rawmodel; import org.lwjgl.util.vector.vector2f; import org.lwjgl.util.vector.vector3f; public class objloader { public static rawmodel loadobjmodel(string filename, loader loader){ filereader fr = null; try { fr = new filereader(new file("res/"+filename+".obj")); } catch (filenotfoundexception e) { system.err.println("couldn't load file!"); e.printstacktrace(); } bufferedreader reader = new bufferedreader(fr); string line; list<vector3f> vertices = new arraylist<vector3f...
Read more

spring - SubProtocolWebSocketHandler - No handlers -

-
i have ugly error during deploying spring app on jboss. 18:11:16,025 error [org.apache.catalina.core.containerbase.[jboss.web].[default-host].[/consumer]] (msc service thread 1-7) exception sending context initialized event listener instance of class org.springframework.web.context.contextloaderlistener: org.springframework.context.applicationcontextexception: failed start bean 'subprotocolwebsockethandler'; nested exception java.lang.illegalargumentexception: no handlers @ org.springframework.context.support.defaultlifecycleprocessor.dostart(defaultlifecycleprocessor.java:176) [spring-context-4.1.3.release.jar:4.1.3.release] @ org.springframework.context.support.defaultlifecycleprocessor.access$200(defaultlifecycleprocessor.java:51) [spring-context-4.1.3.release.jar:4.1.3.release] @ org.springframework.context.support.defaultlifecycleprocessor$lifecyclegroup.start(defaultlifecycleprocessor.java:346) [spring-context-4.1.3.release.jar:4.1.3.release] @ org.s...
Read more

methods - python can't use function in submodule -

-
my folder structure this: pythonstuff/ program.py modulea/ __init__.py foo.py bar.py here's code bar.py : def hello(): print("hello, world!") here's code program.py : #!/usr/bin/env python3 modulea import bar bar.hello() i run $ python3 program.py somehow error: file "program.py", line 3, in <module> bar.hello() attributeerror: 'module' object has no attribute 'hello' edit: __init__.py file empty. edit2: after trying realized had bar.py in root directory contained hello() method. bar.py in modulea/ directory empty. add it import os __all__ = [] module in os.listdir(os.path.dirname(__file__)): if module != '__init__.py' , module[-3:] == '.py': __all__.append(module[:-3]) the init .py files required make python treat directories containing packages; done prevent directories common name, such string, unintentionally hiding v...
Read more