// the find
shafiab/HashtagCashtag
My Insight Data Engineering Fellowship project. I implemented a big data processing pipeline based on lambda architecture, that aggregates Twitter and US stock market data for user sentiment analysis using open source tools - Apache Kafka for data ingestions, Apache Spark & Spark Streaming for batch & real-time processing, Apache Cassandra f or storage, Flask, Bootstrap and HighCharts f or frontend.
A 2016-era Insight Data Engineering fellowship project that builds a lambda architecture pipeline over Twitter and NYSE/NASDAQ stock data to correlate tweet sentiment with stock price movement. It wires together Kafka, Spark batch jobs, Spark Streaming, Cassandra, and a Flask frontend. The live site is long dead and the Twitter API it depended on no longer exists in this form.
The Cassandra schema design shows real thought — partitioning by ticker with clustering on time components is correct for the query patterns, and the TTL workaround for the rolling-count dashboard is a honest solution to a genuine connector limitation. The lambda architecture is implemented properly: batch and speed layers actually cover the same aggregations rather than splitting concerns arbitrarily, which is a common mistake in tutorial projects. The pipeline diagram and README explain architecture decisions with enough detail that you can understand why choices were made, not just what they are. Azkaban for batch scheduling is a reasonable pick given the parallelization and retry requirements.
Sentiment analysis is a keyword +1/-1 counter — described accurately in the README as 'very simple', but this makes the correlation charts largely meaningless noise dressed up as insight. The Twitter streaming API dependency is dead; this project cannot run as described. The stock data source (netfonds.no) is also gone. The speed layer's use of `updateStateByKey` without checkpointing discussion is a reliability gap — that state is lost on driver failure. No tests anywhere in the repo.