Orca: Ocean Significant Wave Height Estimation with Spatio-temporally Aware Large Language Models

Intro

two categories of SWH estimation:

1. traditional
2. machine learning-based methods

traditional methods：

1. weak1: too slow to estimate in time
2. weak2: Difficult to adapt to additional impact factors, limits opportunities for performance improvement.

ML-based methods:

Figure omitted during migration: image-20241208012411958.png was referenced locally but the asset file is not present in this repo.

1. challenge1: data is too sparse.
2. challenge2: the wave variations have strong spatio-temporal correlations, current methods fail to capture this intricate relationships and discern the wave dynamic patterns implied in the data.

LLM?

1. few-shot performance
2. tackle the issue of data sparsity in SWH estimation

Contributions

1. Propose a LLMs with spatio-temporal awareness -> SWH estimation
2. invent a prompt templates and prompt embedding module -> To tackle the data sparsity
3. A novel spatio-temporal aware encoding moudle -> Enhance the spatio-temporal reasoning capabilities of LLM

Preliminaries

Significant Wave Height: SWH is defined as the average height of the top 1/3 waves

Buoy-based Data(X): X in R(FxMxT), F features collected by M stationary oceanic buoys over T continuous time intervals

Grid-based SWH: GSWH,Y in R(KxJxT), records the average SWH, each grid area over T continuous time intervals, where K and J indicate the rows and columns, respectively.

Problem Definition

Given the buoy-based data X over T continuous time intervals, to estimate the GSWH values Y of T continuous time intervals.

Methodology

Prompt Designing and Encoding

Divide the prompt into five components:

1. Actor
2. Information
3. Target
4. Features
5. Data

The word embedding layer of LLMs takes input as the structured prompt P = {p1, p2, … , pE }, resulting in P = {p1, p2, . . . , pE }, where E is the length of the prompt, pi ∈ RD is the embedded vector of the corresponding token pi , where D is the dimension of the semantic space of LLMs

Following P-Turning, randomly generate a fixed number of soft-prompt Q = {q1,q2,…,qR},R is the length of soft-prompt, after embedding, obtain Q = {q1,q2,...,qR}.

For Q, we use a mlp(two linear) to Modeling the dependency relationships between different embeddings, obtaining Hq.

Final prompt is concate Hq and P , get Hprompt=[Hq;P] in R: (R+E)XD

Spatio-temporal Encoding

Spatial Encoding

(challenge2): Any maritime coordinate is inherently influenced by the wave movements in its vicinity, but LLMs has limited spatial reasoning capabilities.

The locations of buoys are denoted as G = {(ui, vi)}, i in 1 to M,where u and v indicate the row and column grid indices for the i-th buoy.

Employ Z-order curve, get Z = Zorder(G), Z ∈ R(MxA), M is the number of buoys and A is one-dimensional binary embedding.

Subsequently, Use a linear on Z, get Hloc ∈ R(MXD).

Temporal Ecoding

the observed data X of buoys is segmented into overlapping patches C = {X(i)}, i from 1 to S, X(i) ∈ R(FxMxL), S = (T-L)/W +2, W, L are the stride and the patch length.

Obtain Htemp in R(SxFxMxD) by a linear.

Final Prompt

Hinput = [Hprompt;Hloc;Htemp] ∈ R(IxFxMxD)，and I = R + E + 1 + S

LLMs Fine-tunning

Hllm = LLM(Hinput) in R(IxFxMxD), Hpool = AvgPool(Hllm) in (IxMxD)

use a linear and a Reshape Function, Let Hpool transform to Y’s shape

Optimizing with Physical Regularization

Loss: MSE

1. l1:SWH values from buoys and the estimated GSWH values.
2. l2:data from numerical model and estimated GSWH values.

loss = l1+ axl2

Experiments

Datasets

1. the Buoy-based data is from the National Data Buoy Center dataset (NDBC)
2. The corresponding numerical model data is from the Global Wave Database (GWD), provided by the Estuary and Coastal Laboratory

Figure omitted during migration: image-20241208175257340.png was referenced locally but the asset file is not present in this repo.

Figure omitted during migration: image-20241208175306375.png was referenced locally but the asset file is not present in this repo.

Summary