Overview

Orbits is a tool to gather earth orbitology and project coverage predictions for specified satellites. It can produce just TLEs, or TLEs and rasters.

PosAtTime is a specialized module that uses Orbits submodules to either print satellite (lon, lat) at a specified seed time, or produce a raster of locations for a list of seed times.

Installation Guidelines

Code from this project will be installed and set up via the following repo install scripts: **busboy_install** This install script is designed for a CentOS environment.

apikeys.json must be provided in orbits/conf/apikeys.json for any Space-Track queries to work. See conf/example_apikeys.json

The install actions can be used to inspire installation in another environment, but no guarantees are made as to functionality.

Execution

Before execution activate the orbits virtual environment, either by calling source orbits/power_on then deactivating with source orbits/power_off, or by using the code included in "shell_header".

Orbits

Orbits multisat execution can be done by:

Using a satcatid list in the json properties file, and calling via Orbits.p. Most efficient, as Orbits.py parallelizes on satcatid
Using a single satcatid in the json properties file
Constructing some other script to call Orbits.py, TLEGen.py, or SatRaster.py with option argument -sID <satcatid> sequentially for each satellite. Less efficient, as Orbits.py parallelizes on satcatid

Below are a sample of common code paths to inspire scripting:

To output a raster for each of the default satellites (in default_orbits_properties.json) for a specified time range (t0 to t1), see DTHelpers for time formatting conventions("%Y-%m-%d-%H-%M-%S-%f"):

.orbits/src/Orbits.py -t0 "2013-01-01-00-00-00-000000" -t1 "2013-01-10-00-00-00-000000" -time_format "hyph_str" -sID 25919

To source TLEs, then output rasters for the default time period for all satellites in the properties file. Default properties file: default_orbits_properties.json, or specify property file (absolute path, using -MC option):
- Default properties: ./orbits/src/Orbits.py
- Custom properties: ./orbits/src/Orbits.py -MC </absolute/path/custom_orbits_properties.json>

To source TLEs, then output a raster for the default time period (in default_orbits_properties.json) for a specific satellite:

./orbits/src/Orbits.py -sID <satcatid>

To get the TLEs for a satellite for a specified time:

./orbits/src/Orbits.py -RT "tle_only" -t0 "2013-01-01-00-00-00-000000" -t1 "2013-01-10-00-00-00-000000" -time_format "hyph_str" -sID 25919

To output raster for a specified time, either using TLEs in a given folder (NOTE: TLE filename must match expected, or be specified as an -MIF file):
- Path to TLEs that match provided time: ./orbits/src/Orbits.py -RT "raster_only" -MTD <path/to/TLEs> -t0 "2013-01-01-00-00-00-000000" -t1 "2013-01-10-00-00-00-000000" -time_format "hyph_str" -sID 25919
- Absolute path to specific TLE to make raster from: ./orbits/src/Orbits.py -RT "raster_only" -MIF </absolute/path/to/TLE.txt> -t0 "2013-01-01-00-00-00-000000" -t1 "2013-01-10-00-00-00-000000" -time_format "hyph_str" -sID 25919

PosAtTime

PosAtTime is a standalone file, and must be run for each satcatid in question. There are two code routes:

Provide a single seed time: with TLE (as string or file) OR without TLE (nearest TLE will be sourced from SpaceTrack)
- Output is a printed (lon, lat)
Provide a list of seed times: with TLEs as file OR without TLE (TLEs for the time range will be sourced from SpaceTrack)
- Seed time file 1 time per line, of the format 2008-11-10-17-53-59-000317 (YYYY-MM-DD-hh-mm-ss-µs)
- TLEs must be in new->old order (default TLEGen output style)
- Seed time list will be sorted into appropriate order and de-duplicated (no rendundant times)
- Output is a raster

To print the position of a satellite at a specified time with a user defined TLE, or sourcing the nearest TLE if it is not supplied:
- Sourcing the nearest TLE to seed_time: ./src/PosAtTime.py -sID 25919 -ST "2014-11-18-00-00-00-000000"
- Providing the reference TLE as a file: ./src/PosAtTime.py -sID 25919 -ST "2014-11-18-00-00-00-000000" -TLEF "../../test/test_data/25919_2014-11-17-19-34.txt"
- Providing the reference TLE as a file, with a specified path: ./src/PosAtTime.py -sID 25919 -ST "2014-11-18-00-00-00-000000" -TLEF "25919_2014-11-17-19-34.txt" -MID "\<path/to/input_files\>"
- Providing the reference TLE as a string (string must be quoted with 2 lines for tle, 3 lines for 3le): ./src/PosAtTime.py -sID 25919 -ST "2014-11-18-00-00-00-000000" -TLE "1 25919U 99051A 14321.81546994 .00001026 00000-0 20257-3 0 862 2 25919 098.1009 034.1082 0001676 069.4052 290.7309 14.64484712809793"

To produce a raster of the position(s) of a satellite at a specified list of times with user defined TLEs, or sourcing the nearest TLE if it is not supplied:
- Sourcing the TLEs within the time range provided in the list: ./src/PosAtTime.py -sID 25919 -STF "\<times_file\>"
- Providing the reference times and TLEs as a file: ./src/PosAtTime.py -sID 25919 -STF "\<times_file\>" -TLEF "25919_2009-01-01-01-07-16-800000_to_2010-12-31-21-45-34-900000_tle.txt"
- Providing the reference times as a file, sourcing TLEs, and specifying input and output directories: ./src/PosAtTime.py -sID <satcatid> -STF "\<times_file\>"" -MID "<path/to/input_files>" -MOD "<path/to/output/raster>"

Time

Precision timing is critical to drawing conclusions from satellite positioning, see details in other sections. When possible, standardize to UTC. Any usage of the special time **"now"** will set time to the current UTC time, this includes output directory naming.

This module utilizes Python datetime objects, and a library of conversion functions is provided in DTHelpers.py

In terms of configuration times can be supplied in two ways, with a consistent "time_format". Mixing these two routes will raise exceptions:

"t0" and "t1". Start and end times can be provided in several formats via JSON file or option argument as a string. t1(end time) > t0 (start time)
"t0", "t_range", "t_range_unit", "t_forward". In this way, t0 and t1 will be calculated either backward or forward from the initial seed time "t0", for the specified length of time.

Manual time can be set by option arguments with the format -t0 <t0> -t1 <t1> -time_format <time_format>

PosAtTime accepts one time format, so "time_format" is not specified. It is 2008-11-10-17-53-59-000317 (YYYY-MM-DD-hh-mm-ss-ms or Y-m-d-H-M-S-f)

Accuracy

Accuracy of this module should be within 1 raster cell (default 0.1 deg).

Type conversion and accumulating float errors are the main sources of error, and within this tolerance, errors are rounded to fit within the raster or (lon, lat) bounds. Errors beyond this tolerance will raise exceptions.

TLEs are another source of error, and therefore the closest TLE to a given time will be used. TLEs > 1 year from the time of a calculation will not be used.

Exceptions

Every effort has been made to raise meaningful exceptions that can be traced back to lines of code and relevant modules for debugging. Errors will be of the form "Traceback (most recent call last): ... Exception".

Since Orbits generates subprocesses, one of these may fail while the others succeed. Check output and exception messages accordingly.

In submodule (TLEGen, SatState, SatRaster) and PosAtTime execution, a single exception will terminate the program.

Path Formatting

Note that paths beginning with '/' will be interpretted as absolute within this module. If a path should be relative, drop the beginning slash. It will be added when paths are joined. This is particularly relevant when specifying manual input and output files.

Default input and output paths are: orbits/output/<YYYY-MM-DD> Any changed to output/input paths should use option arguments.

Output Naming Conventions

Orbits

Format can be overridden by supplying manual output via command line argument or config file.

Default Orbits file format:

{satcatid}_{time_range}_{freq}.{extension}

{satcatid}

the NORAD satcatid

{time_range}

{t0}_to_{t1}
{t0} and {t1} are timestamps
"%Y-%m-%d-%H-%M-%S-%f"
2013-11-10-17-53-59-000317

{freq}

the sample frequency for satellite locations
units in seconds, "sec"

{extension}

TLEs: ".txt"
SatRaster/GeoTiff: ".tif"

3LE/TLE output

25919_2013-11-10-00-00-00-000000_to_2014-11-10-00-00-00-000000_5sec.txt 25919_2013-11-10-17-53-59-000317_to_2014-11-10-17-53-59-000317_5sec.txt

SatRaster output

25919_2013-11-10-00-00-00-000000_to_2014-11-10-00-00-00-000000_5sec.tif 25919_2013-11-10-17-53-59-000317_to_2014-11-10-17-53-59-000317_5sec.tif

PosAtTime

Formats are different to distinguish from Orbits output.

Seed Times

Input only, provided by user or made programatically elsewhere, can have different naming convention if specified by option argument. {satcatid}_times.txt 25919_times.txt

TLEGen output

TLEs (if TLEs over multiple seed times sourced because not provided). If user is providing TLEs, they must be in new->old order (TLEGen default format) {satcatid}_{t0}_to_{t1}_tle.txt 25919_2009-01-01-01-07-16-800000_to_2010-12-31-21-45-34-900000_tle.txt

SatRaster output

A raster file {satcatid}_{t0}_to_{t1}_times.tif 25919_2009-01-01-01-07-16-800000_to_2010-12-31-21-45-34-900000_times.tif

Packaging

working within isolated python instance has numerous benefits
installation will configure this virtual environment(aka venv, virtualenv)
pip will handle most python module installation via install_requirements.txt
there are some custom installs within the project installation scripts

Code

In alphabetical order, see src for main source code

bin

virtual environment container folder, advanced users only

conf

Configurations will be set in the following priority hierarchy: Module constructors > option arguments > properties JSON > module defaults An option argument can be supplied to override properties JSON value for the same parameter. JSON files can be versioned, and are recommended. Many properties can only be set via JSON.

JSON

portable data format used in this library to hold configurations
recommended validation of JSON configuration files with a tool like JSONLint, JSONView: Firefox or Chrome
Note: JSON defines numbers as '0' or beginning with '1-9', so leading '0's are not permitted
JSON also does not allow comments

conf/apikeys.json

holds required user-specific apikeys for TLE sourcing
format either user/password (space-track convention) or single key (wired in for other potential TLE sources)
see conf/example_apikeys.json for examples of this JSON structure.

conf/cron_orbits_properties.json

holds module configurations for nightly cron example
see orbits_ex.sh for use

conf/orbits_default_properties.json

holds module configurations with expected values
holds an example set of satellites for queries
single satellites can be queried using the command line as well
properties can be directly configured in this file, or copied to a new/renamed file, and then specified through command line options ('-MC' or '–man_conf')

conf/example_apikeys.json

examples of the two expected formats for apikeys

conf/example_orbits_properties.json

example of possible values for configuring Orbits, intended as a companion to other JSON for reference
contents can not be run as is, they are a list of configurable properties and examples of options for a given field
arrays & lists should be interpreted as options from which one must be chosen. [0, 1] indicates a boolean flag property. Exception is satcatid, supplied as a list.
- eg: "tle_format": ["tle", "3le"], tle_format is either "tle" or "3le", both are listed to show which can be chosen
strings & numeric values show literal params that can be entered
dictionaries/maps show examples of properly formatted items, or lists from which one must be chosen
Property details below:
- "out_dir", dict, choose one ["default", "relative", "absolute"] and supply corresponding value ["default", <relative or="" absolute="" path>="">]
- "out_date_dir", boolean flag, build optional subdirectory <YYYY-MM-DD>, UTC based
- "in_dir", dict, choose one ["default", "relative", "absolute"] and supply corresponding value ["default", <relative or="" absolute="" path>="">]
- "in_date_dir", boolean flag, build optional subdirectory <YYYY-MM-DD> in "out_dir", UTC based
- "f_out", dict, choose one ["default", "custom"] and supply corresponding value ["default", <filename with="" or="" without="" extension>="">, <relative or="" absolute="" path="" to="" filename>="">] }
- "f_in", dict, choose one ["default", "custom"] and supply corresponding value ["default", <filename with="" or="" without="" extension>="">, <relative or="" absolute="" path="" to="" filename>="">] }
- "satcatid": list, one or more numeric satcatid
- "sat_name": dict, one or more {"<satcatid>": "<satname>"} pairs, should correspond to satcatid, must both be strings
- "t_range_unit": list, choose one ["day", "days", "year", "years"], value of 't_range_unit' in <x> <t_range_unit>, e.g. 1 day
- "t_range" : numeric, value of 'x' in <x> <t_range_unit>, e.g. 1 day
- "t_forward": boolean flag, is the range forward or backward in time from "t0"
- "time_format": string, choose one ["dtime", "epoch_utc", "epoch_local", "time_str", "manual_time", "ts_tuple", "hyph_str"], correspond to conversions in DTHelpers.py
- "t0": string, epoch(float), dict, or list; must correspond to the "time_format", choose one ["now", 1410449241.01 (example of epoch), (example of a time dict) {"year": 2014, "month": 9, "day": 11, "hour": 15,"minute": 27,"second": 21, "microsecond": 8236}, (example of a time list)[ 2014, 9, 11, 15, 27, 21, 8236]],
- "t1": same as t0, must be later in time
- "verbose": boolean flag, optional verbose messaging
- "query_sat_name": boolean flag, optional query to space-track to validate satellite name
- "tle_format": string, specify TLE format, choose one ["tle", "3le"]
- "tle_provider": string, specify provider, new providers require coding in TLEGen.py, choose one ["space-track"]
- "tle_sesh_timeout": numeric, session timeout value in seconds
- "multi_tle": boolean flag, option to source most recent TLE for provided t0-t1 range or all TLEs from this range, recommend multi-tle
- "no_empty_tle": boolean flag, option to source a single closest TLE to range t0-t1 if there are no TLEs for that range in the initial query
- "sec_sample_rate": numeric, calculate satellite position at this second-based rate
- "validate_time": boolean flag, option to prevent using TLEs beyond SatState.MAX_T_DELTA which can return invalid positions, MAX_T_DELTA = 364 days
- "allow_missing_tle_checksum": boolean flag, option to allow TLEs missing the checksum, (for example 25919 in December 2001), will calculate checksum and attempt to proceed with repaired TLEs
- "pixel_res": numeric, the resolution of a pixel in SatRaster output, currently square pixels with SatRaster.x_res = SatRaster.y_res = pixel_res
- "x_dim": numeric, the horizontal dimension of SatRaster output, default to 3600
- "y_dim": numeric, the vertical dimension of SatRaster output, default to 1800
- "x0": numeric, the top left x origin, default -180
- "y0": numeric, the top left y origin, default 90
- "raster_driver": string, the GDAL raster driver, currently must be default "GTiff"
- "num_bands": numeric, number of bands in SatRaster output, default 1, multiple bands have not been tested
- "stream": boolean flag, option to stream positions to SatRaster, streaming improves memory usage and speed and is enabled by default
- "do_raster_stats": boolean flag, option to calculate raster statistics (min, max, range, median, mean, std_dev, variance, sum), useful for testing, can be further used for quantitative analysis of stats by lat/lon, or whole raster
- "validate_pos": boolean flag, option to validate that satellite position is not NaN, an artifact of bad TLEs, de-orbiting, etc.
- "route": string, Orbits.py code route, choose one ["tle_raster", "tle_only", "raster_only"]

conf/install_requirements.txt

versioned project requirements that can be installed automatically
reference file, do not edit

conf/requirements.txt

versioned, reflects all expected packages within the virtualenv
reference file, do not edit

conf/my_requirements.txt

used to indicate locally installed project packages
programmatically generated by install script & checked by power_on

conf/setup_orbits

internal script called by power_on to ensure packages installed and up to date

conf/updated

used to indicate last time of locally updated packages

include

virtual environment container folder, advanced users only

lib

virtual environment container folder, advanced users only

man

virtual environment container folder, advanced users only

orbits_ex.sh

script to show examples of Orbits.py with cron job, default job, tests, or internet tests
- ./orbits_ex.sh default
- ./orbits_ex.sh cron
- ./orbits_ex.sh tests
- ./orbits_ex.sh tests internet

orbits2raster.sh

script to show example of calling Orbits and merging resulting rasters.

output

holds output in various sub-directories, with the specified file naming patterns

power_on

script to intialize Orbits virtual environment, call from within the orbits/ directory source power_on

power_off

script to deactivate Orbits virtual environment, call from within the orbits/ directory source power_off

README.md

this file, documentation of the Orbits module

shell_header

a stub script that encapsulates the function of power_on and power_off for use in scripts that run from arbitrary locations

src

Holds source code
Highest level interaction is with Orbits.py
Data flow within a call to Orbits.py: TLEGen -> SatState -> SatRaster (may happen multiple times if multiple satellites)
All modules can be used in isolation in some capacity or another, with the exception of OrbitsBase, which has no standalone functionality. SatState is mainly an internal module.

src/drive_tle_raster.sh

script called by Orbits.py via xargs to parallelize Orbits submodules along satcatid
not for end use, use Orbits.py

src/DTHelpers.py

Library of date/time manipulation helper functions. Time-based parameter arguments in other classes should be python datetime objects unless otherwise specified.

src/Orbits.py

Main driver class for this section of satelliteMetadata.

src/OrbitsBase.py

Base class for all Orbits modules. Includes exception classes, base functionality, and common module imports for simplicity.

src/PosAtTime.py

Standalone module to output satellite location (lon, lat) at a seed time OR produce a raster of locations from a list of seed times, optionally sourcing the closest TLE(s) or using provided TLE(s)

src/SatRaster.py

Ingests satellite positions and emits geotiff rasters.

src/SatState.py

Ingests tle orbitology and emits satellite positions.

src/TLEGen.py

Ingest satellite catalogue ID and emits TLE orbitology.

test

holds test code & test data
These scripts test the corresponding source file, tests with "internet" in them require internet connectivity.

test/test_data

holds test data

test/test_output

holds test output

Command line options

AKA "args" or "option args" can always be accessed by <ModuleName.py> --h eg: python Orbits.py --help or python Orbits.py --h

Common args (OrbitsBase.py): Apply to all modules except PosAtTime '-sID', '–satcatid', help='NORAD_CAT_ID, only one'
Numeric Satellite Catalog Number specified by NORAD '-MC', '–man_conf', help='manual configuration file, absolute path'
Manual time can be configured using three option arguments -t0 <t0> -t1 <t1> -time_format <time_format> '-t0', '–t0', help='set t0 manually, must be string', type=str) '-t1', '–t1', help='set t1 manually, must be string', type=str) '-time_format', '–time_format', choices=["epoch_utc", "epoch_local", "time_str", "hyph_str"], help='set time_format_manually, must be string'
Note limited set of time_format values '-MOD', '–man_output_dir', help='manual output directory, absolute path'
Note that output and input directories need to connect between TLEGen and SatRaster, so use -MTD OR -MRD if custom output directories desired from Orbits, otherwise from Orbits -MOD and -MID must be the same
default I/O directories are orbits/output/<YYYY-MM-DD>/ where the date portion is used unless the JSON configurations turn it off '-MID', '–man_input_dir', help='manual input directory, absolute path' '-MOF', '–man_output_file', help='manual output file, relative (absolute if begins with "/"' '-MIF', '–man_input_file', help='manual input file, relative (absolute if begins with "/"'

Orbits.py specific args: '-RT', '–route', help=('set Orbits execution routine,["tle_raster", "tle_only", "raster_only"]' '-MTD', '–man_tle_dir', help='manual input TLE directory, absolute path'
Set custom TLEGen output and correspondingly the -MID for SatRaster '-MRD', '–man_raster_dir', help='manual input/output raster directory, absolute path'
Note that output and input directories need to connect between TLEGen and SatRaster, so use -MTD and -MRD if custom output directories desired from Orbits, otherwise from Orbits -MOD and -MID must be the same
**DO not use -MTD and -MRD together, they both set the SatRaster input directory(-MID), call SatRaster directly if this behavior is desired

TLEGen specific args: '-tF', '–tle_format', help='tle_format, ["tle", "3le"]'

SatState specific args: None

SatRaster specific args: None

PosAtTime specific args: '-sID', '–satcatid', help='NORAD_CAT_ID, only one', type=str)
a single NORAD satcatid, for multisat, call PosAtTimes for each satellite separately '-ST', '–seed_time', help='seed_time', type=str)
a single seed time string of "hyph_str" format: 2008-11-10-17-53-59-000317 (YYYY-MM-DD-hh-mm-ss-ms or Y-m-d-H-M-S-f) '-STF', '–seed_time_file', help='seed_time_file', type=str)
a file of seed times, one per line '-TLE', '–tle_str', help='TLE or 3le string, lines separated by \n, \r\n, or \r', type=str)
must be quoted because of newlines '-TLEF', '–tle_input_file', help='TLE or 3le file as produced by TLEGen, lines separated by \n, \r\n, or \r', type=str)
must be in new->old order (TLEGen default format) '-MOD', '–man_output_dir', help='manual output directory, absolute path', type=str) '-MID', '–man_input_dir', help='manual input directory, absolute path', type=str)
Note that output and input directories need to connect if TLEs are being produced by PosAtTime
default I/O directories are orbits/output/<YYYY-MM-DD>/ where the date portion is used unless the JSON configurations turn it off '-MRF', '–man_raster_file', help=('manual output file, relative (absolute if begins with "/"'), type=str)
if a different output name is desired

External Resources

space-track.org

TLE provider used for beta development

API documentation

session cookies persist for 2 hours
Throttling will occur beyond 20 requests/min

Two Line Element (TLE) Format

also described well by TLE format and Celestrak: TLE

Satellite Catalog

GDAL

GDAL Raster Formats(http://www.gdal.org/formats_list.html)

CelesTrak

Another popular TLE provider

Python Libraries

The following libraries are either in use, or worth noting for development purposes.

os & sys

core python modules for interacting with host OS and python interpreter

shutil

core python module for high-level interactions with files

collections

core python module for high performance datatypes

subprocess

core python module for shell command interaction

multiprocessing

core python module for a "threading" interface
not used here for parallel processing, xargs is instead for portability

json

core python module for JSON

argparse

core python module for command line args

time & datetime

core python modules for handling time & date
the pyephem module requires specific naive datetime values, so support functions have been implemented from the Salty Crane Cheat Sheet and other references

dateutil