The supporting coursework behind the e-commerce analytics project: regional bestsellers, product conversion, Top-10 products and user-item matrices with Spark RDD.
~2M
behaviour records
05
funnel stages
~30%
less shuffle overhead
01 / Overview
What this package actually contains.
The Spark work converted raw user-visit logs into product and funnel signals. The implementation uses RDD transformations for aggregation, joins and a user-item click matrix, with broadcast variables reducing repeated lookup work.
Objectives
01Aggregate approximately two million behaviour records across products, regions and funnel stages.
02Identify the largest drop-off points between browsing, adding to cart and checkout.
03Prepare a scalable user-item representation for collaborative-filtering recommendations.
Data flow
From raw records to a usable service.
The pipeline is the main story: ingestion, storage, processing and serving.
01→
Visit logs
User actions, timestamps, city and clicked product identifiers.
02→
City/product joins
Dimension tables enrich raw events with area and product metadata.
03→
RDD aggregation
reduceByKey, full outer joins and top-N sorting.
04→
Funnel + matrix
Click/order/pay rates and CoordinateMatrix output.
05
Recommendation input
Sparse user-item matrix ready for collaborative filtering.
02 / Package highlights
What this work demonstrates.
Implements four analytics modules over user-visit action logs and city data.
Connects distributed transformations to concrete e-commerce business questions.
Includes the full report and source archive behind the CV project.
03 / Implementation
Where the engineering complexity sits.
01
Funnel analysis
Click, order and payment counts were calculated independently and joined by product. Full outer joins kept products that appeared in only one stage, which matters when diagnosing conversion leakage. Safe division prevented zero-click products from producing invalid rates.
The funnel used five stages from browse to add-to-cart, order and payment.
Sorting by click volume made the highest-impact drop-off points visible first.
Missing order or payment stages remained explicit rather than being dropped silently.
02
Shuffle and lookup control
The user and product index maps were collected once and broadcast to executors before constructing MatrixEntry records. This removed a repeated distributed join from the matrix-building path and contributed to the reported reduction in shuffle overhead.
Broadcast variables are appropriate because the dimension maps are relatively small.
CoordinateMatrix stores only observed user-product clicks.
The matrix can be passed to a later recommendation model without rebuilding the text pipeline.
03
What the project demonstrates
The project combines business-facing analysis with distributed engineering. It is not only a dashboard exercise: the same RDD pipeline produces funnel metrics, product rankings and a machine-learning input representation.
04 / Key code
Code that defines the pipeline.
Only verified source excerpts and pseudocode taken from the project papers are shown.
Verified project codeScala
Product conversion funnel
Original Scala excerpt from the Spark coursework.
ProductConversionAnalysis.scala
val clickRDD = data
.filter(fields => fields(7) != "-1")
.map(fields => (fields(7), 1))
.reduceByKey(_ + _)
val orderRDD = data
.filter(fields => fields(9) != "\N")
.flatMap(fields => fields(9).split(",").map(pid => (pid, 1)))
.reduceByKey(_ + _)
val payRDD = data
.filter(fields => fields(11) != "\N")
.flatMap(fields => fields(11).split(",").map(pid => (pid, 1)))
.reduceByKey(_ + _)
val resultRDD = clickRDD
.fullOuterJoin(orderRDD)
.fullOuterJoin(payRDD)
.mapValues { case (coOpt, payOpt) =>
val (clickOpt, orderOpt) = coOpt.getOrElse((None, None))
(
clickOpt.getOrElse(0),
orderOpt.getOrElse(0),
payOpt.getOrElse(0)
)
}
Verified project codeScala
Broadcast-backed user-item matrix
Original Scala excerpt showing index broadcasting and CoordinateMatrix construction.
UserProductClickMatrix.scala
val userProdCount = raw
.map(up => (up, 1))
.reduceByKey(_ + _)
.map { case ((user, product), count) =>
((user, product), count.toDouble)
}
val userIndex = raw.map(_._1).distinct().zipWithIndex().collectAsMap()
val productIndex = raw.map(_._2).distinct().zipWithIndex().collectAsMap()
val bUserIndex = sc.broadcast(userIndex)
val bProductIndex = sc.broadcast(productIndex)
val entries = userProdCount.map {
case ((user, product), count) =>
new MatrixEntry(
bUserIndex.value(user),
bProductIndex.value(product),
count
)
}
val matrix = new CoordinateMatrix(entries)
Operational notes
Full outer joins preserve products that only appear in one funnel stage.
Broadcast maps eliminate repeated user/product lookups in the matrix-building stage.
CoordinateMatrix is a natural fit for sparse click data and downstream collaborative filtering.
Limitations / next steps
A production recommendation system would require offline evaluation against held-out interactions.
The pipeline should be parameterised by date partitions rather than local file paths before deployment.
05 / Downloads
Files in this folder.
Open the original report, presentation or source archive. PDF cover pages with private details were removed.
PDF
Spark analytics report
中文 · 23 pages · 1.1 MB
Sanitized 23-page report with architecture, code and result screenshots.
