1. Advana and Jupiter


Advana: DoD analytic environment https://advana.data.mil/
               Databricks, DataRobot, Qlik, Tableau, EDL, and Catalog
Jupiter: https://jupiter.data.mil/
               Navy’s cloud-based data lake and analytics environment within ADVANA
               (ARES for the Army and BLADE for the Air Force)
                AWS GovCloud and operational on NIPR, SIPR and JWICS networks
Service desk: https://advana.data.mil/landing/servicedesk
ADVANA knowledge: https://wiki.advana.data.mil/display/SDKB/Get+Help
Upcoming training: https://wiki.advana.data.mil/display/SDKB/Monthly+Training+Series
ADVANA training: https://wiki.advana.data.mil/display/SDKB/Get+Training
                              https://wiki.advana.data.mil/display/SDKB/Beginner+Trainings
ADVANA office hours: https://wiki.advana.data.mil/display/SDKB/Office+Hours+Calendar
EDL for external data: https://edl.advana.data.mil/
Support team email:




2. Spatial data forecasting
NASA data: https://espo.nasa.gov/atom
The data downloaded from NASA is flight data to measure atmospheric tomography mission(pollution environmental factors). Flights will occur in each of 4 seasons over a 4-year period. The deployment originated from the Armstrong Flight Research Center in Palmdale, California, flew north to the western Arctic, south to the South Pacific, east to the Atlantic, north to Greenland, and returned to California across central North America.



Flight1 with 3380 rows. The flight starts from 14:33:10 and ends at 23:56:30
Made the forecasting more challenging that the rows of 1660~1680 is replaced as zeros and see if advanced models can predict the missing rows in the middle of the flight 

## # A tibble: 21 × 12
##    Index    FlightNum     Date Start    End        Lat  Long   Alt Press  Temp
##    <chr>        <dbl>    <dbl> <time>   <time>   <dbl> <dbl> <dbl> <dbl> <dbl>
##  1 A1_01660         1 20160729 19:09:40 19:09:50  5.17 -121.  192.  992.  298.
##  2 A1_01661         1 20160729 19:09:50 19:10:00  5.15 -121.  193.  992.  298.
##  3 A1_01662         1 20160729 19:10:00 19:10:10  5.14 -121.  194.  992.  298.
##  4 A1_01663         1 20160729 19:10:10 19:10:20  5.13 -121.  195.  992.  298.
##  5 A1_01664         1 20160729 19:10:20 19:10:30  5.12 -121.  194.  992.  298.
##  6 A1_01665         1 20160729 19:10:30 19:10:40  5.10 -121.  194.  992.  298.
##  7 A1_01666         1 20160729 19:10:40 19:10:50  5.09 -121.  195.  992.  298.
##  8 A1_01667         1 20160729 19:10:50 19:11:00  5.08 -121.  194.  992.  298.
##  9 A1_01668         1 20160729 19:11:00 19:11:10  5.07 -121.  194.  992.  298.
## 10 A1_01669         1 20160729 19:11:10 19:11:20  5.06 -121.  194.  992.  298.
## # ℹ 11 more rows
## # ℹ 2 more variables: groundDst <dbl>, time <dbl>
## # A tibble: 21 × 12
##    Index FlightNum     Date Start    End        Lat  Long   Alt Press  Temp
##    <dbl>     <dbl>    <dbl> <time>   <time>   <dbl> <dbl> <dbl> <dbl> <dbl>
##  1  1660         1 20160729 19:09:40 19:09:50     0     0     0  992.  298.
##  2  1661         1 20160729 19:09:50 19:10:00     0     0     0  992.  298.
##  3  1662         1 20160729 19:10:00 19:10:10     0     0     0  992.  298.
##  4  1663         1 20160729 19:10:10 19:10:20     0     0     0  992.  298.
##  5  1664         1 20160729 19:10:20 19:10:30     0     0     0  992.  298.
##  6  1665         1 20160729 19:10:30 19:10:40     0     0     0  992.  298.
##  7  1666         1 20160729 19:10:40 19:10:50     0     0     0  992.  298.
##  8  1667         1 20160729 19:10:50 19:11:00     0     0     0  992.  298.
##  9  1668         1 20160729 19:11:00 19:11:10     0     0     0  992.  298.
## 10  1669         1 20160729 19:11:10 19:11:20     0     0     0  992.  298.
## # ℹ 11 more rows
## # ℹ 2 more variables: groundDst <dbl>, time <dbl>


3. Modeling Algorithms

1. Kalman filter :It is one of the most popular estimation algorithm for guidance, navigation and control vehicle of aircraft and ships. I believe we, Sea Range, uses Kalman filter for data products for smoothing or other purpose.
2. LSTM : Long short term memory. It is part of neural network, and used for natural language process using input, output and forget gates and which makes it remember values over time interval.
3. Arima/Sarima : The most well-known time series model.
4. Ridge Regressor with Forecasting distance : Autopilot modeling  https://abouttrading.substack.com/p/lets-use-ridge-regression-to-predict

4. DataRobot (ADVANA)

Training and Instruction:https://wiki.advana.data.mil/display/SDKB/DataRobot
https://wiki.advana.data.mil/display/SDKB/DataRobot+Platform+Demo
DataRobot’s time series models employ a data format where the length of the future prediction horizon is not predetermined. Instead, the number of steps to forecast is calculated based on the number of rows available, which directly relates to the size and configuration of the backtesting groups. Consequently, if a 20-step forecast is required, a dataset with only 1159 rows might not provide enough data to execute the algorithm effectively, potentially leading to a prediction sequence of only 10 steps. Blue(actual), Pink(ARIMA/SARIMA),Green(DataRobot), Salmon(Kalman) and Purple(LSTM)

5. How to Use DataRobot with ADVANA (Jupiter)

1. Create an ADVANA account.

2. Request access to the following through ADVANA:
        Databricks
        EDL (External Data Load - https://edl.advana.data.mil/)
        Data Catalog (if your data is in ADVANA Catalog)
        DataRobot

3. Load your data into DataRobot using one of the following methods:
       Option 1: Upload to EDL and then connect to DataRobot.
       Option 2: Directly upload to DataRobot.
       Option 3: Locate data through the Data Catalog and connect to DataRobot.

4. Connect DataRobot to Databricks (Optional):
       Verify the correct cluster is running.
       Create a token within Databricks.
       Establish a connection to the Data Catalog (if applicable).

Can I use DataRobot without importing models to Databricks?
Yes, you can directly upload your data into DataRobot and allow it to run automatically.

Why connect to Databricks?
For implementing forecasting models, combining series of models, model comparison, and exporting outputs to other ADVANA tools.


6. COSMOS(SageMaker)

https://app.cosmos.navy.mil
https://wdibo.navy.mil/esc?id=kb_article&sysparm_article=KB0010300&table=kb_knowledge&searchTerm=Cosmos
What is COSMOS?
COSMOS is a self-service Amazon Web Services (AWS) gov cloud broker designed by and for the Navy.   Cosmos includes AWS services and SageMaker which is an integrated development environment (IDE) for ML with features like code writing, debugging, visualization, and collaboration.