Skip to content Skip to sidebar Skip to footer

What Do The "|" And ">>" Means In Apache Beam?

I'm trying to understand Apache Beam. I was following the programming guide and in one example, they say talk about The following code example joins the two PCollections with CoGro

Solution 1:

The | represents a separation between steps, this is (using p as Pbegin): p | ReadFromText(..) | ParDo(..) | GroupByKey().

You can also reference other PCollections before |:

read = p  | ReadFromText(..)
kvs = read | ParDo(..)
gbk = kvs | GroupByKey()

That's equivalent to the previous pipeline: p | ReadFromText(..) | ParDo(..) | GroupByKey()

The >> are used between | and the PTransform to name the steps: p | ReadFromText(..) | "to key value" >> ParDo(..) | GroupByKey()

Post a Comment for "What Do The "|" And ">>" Means In Apache Beam?"