No relevant resource is found in the selected language.

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Read our privacy policy>Search

Reminder

To have a better experience, please upgrade your IE browser.

upgrade

FusionInsight HD 6.5.0 Product Description 02

Rate and give feedback :
Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document. Note: Even the most advanced machine translation cannot match the quality of professional translators. Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Window

Window

This section describes windows of Flink and provides optimization method. For details about windows, visit the official website at https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/windows.html.

Introduction to Window

Data in a window is saved as intermediate results or original data. If you perform a sum operation (window(SlidingEventTimeWindows.of(Time.seconds(20), Time.seconds(5))).sum) on data in the window, only the intermediate result will be retained. If a custom window (window(SlidingEventTimeWindows.of(Time.seconds(20), Time.seconds(5))).apply(new UDF)) is used, all original data in the window will be saved.

If custom windows SlidingEventTimeWindow and SlidingProcessingTimeWindow are used, data is saved as multiple backups. Assume that the window is defined as follows:

window(SlidingEventTimeWindows.of(Time.seconds(20), Time.seconds(5))).apply(new UDFWindowFunction)

When a data block arrives, it is assigned to four different windows (20/5). That is, the data block is saved as four copies in the memory. When the window lifecycle or sliding period is set to a large value, data will be saved as excessive copies, which is unnecessary.

Figure 4-1 Original structure of a window

If a data block arrives at the 102nd second, it is assigned to windows [85, 105), [90, 110), [95, 115), and [100, 120).

Window Optimization

As mentioned in the preceding, there are excess data copies when original data is saved in SlidingEventTimeWindow and SlidingProcessingTimeWindow. To resolve this problem, the window that stores the original data is restructured, which optimizes the storage and greatly lowers the storage space. The window optimization scheme is as follows:

  1. Use the sliding period as a unit to divide a window into different panes.

    A window consists of one or multiple panes. A pane is essentially a sliding period. For example, the sliding period (namely, the pane) of window(SlidingEventTimeWindows.of(Time.seconds(20), Time.seconds.of(5))) lasts for 5 seconds. If this window ranges from [100, 120), this window can be divided into panes [100, 105), [105, 110), [110, 115), and [115, 120).

    Figure 4-2 Window optimization
  2. When a data block arrives, it is not assigned to a specific window. Instead, Flink determines the pane to which the data block belongs based on the timestamp of the data block, and saves the data block into the pane.

    A data block is saved only in one pane. In this case, only a data copy exists in the memory.

    Figure 4-3 Saving data in a window

  3. To trigger a window, compute all panes contained in the window, and combine all these panes into a complete window.
    Figure 4-4 Triggering a window

  4. If a pane is not required, you can delete it from the memory.
    Figure 4-5 Deleting a window

After optimization, the quantity of data copies in the memory and snapshot is greatly reduced.

Download
Updated: 2019-05-17

Document ID: EDOC1100074548

Views: 3149

Downloads: 36

Average rating:
This Document Applies to these Products
Related Documents
Related Version
Share
Previous Next