the startups.com platform about startups.comCheck out the new Startups.com - A Comprehensive Startup University
Education
Planning
Mentors
Funding
Customers
Assistants
Clarity
Categories
Business
Sales & Marketing
Funding
Product & Design
Technology
Skills & Management
Industries
Other
Business
Career Advice
Branding
Financial Consulting
Customer Engagement
Strategy
Sectors
Getting Started
Human Resources
Business Development
Legal
Other
Sales & Marketing
Social Media Marketing
Search Engine Optimization
Public Relations
Branding
Publishing
Inbound Marketing
Email Marketing
Copywriting
Growth Strategy
Search Engine Marketing
Sales & Lead Generation
Advertising
Other
Funding
Crowdfunding
Kickstarter
Venture Capital
Finance
Bootstrapping
Nonprofit
Other
Product & Design
Identity
User Experience
Lean Startup
Product Management
Metrics & Analytics
Other
Technology
WordPress
Software Development
Mobile
Ruby
CRM
Innovation
Cloud
Other
Skills & Management
Productivity
Entrepreneurship
Public Speaking
Leadership
Coaching
Other
Industries
SaaS
E-commerce
Education
Real Estate
Restaurant & Retail
Marketplaces
Nonprofit
Other
Dashboard
Browse Search
Answers
Calls
Inbox
Sign Up Log In

Loading...

Share Answer

Menu
Data Science: How can I aggregate data from online sources about a specific topic?
SD
SD
Stephen Dearden, Web Dev, Data Mining Expert answered:

This one is tricky because all social media platforms have terms and site rules around collecting personal data. Facebook in particular is touchy about this, so I would not recommend it. Twitter has an API (https://developer.twitter.com/en/docs/tweets/search/overview) that will give you raw data for a small cost, so your hashtag information can be obtained that way. If you are serious about this, then I would identify the top 50-100 businesses that employ students in the area which also keep their pay scale in the range you're looking for. Then using those company's social media pages to identify employees. You are now zeroed on your audience, you just need to fine tune it to filter on the those students that eat out, perhaps by cross referencing Yelp/Google reviews with the employee data obtained.

I would like to re-emphasize here that this sort of data is tricky to obtain as many of the sites containing this data have terms and conditions which discourage collecting personal data.

Are you hoping for a dynamic system that identifies new data sources as they become available? Or will you identify existing sites/pages that give you the data you want and focus on what is generated there? From there, you can obtain the data manually or programatically. Manually, you will actually be visiting those pages on a regular basis, copying and pasting the data there, and then formatting the data to be usable. This is not a scalable solution, but can work if the data sources are limited and you've got the time. Programmatically, you can build small scripts that will make the site requests for you. If you don't program, there are many freelancers/developers that have this skillset and are reasonably priced. There are also 3rd party platforms that are well priced for small scale web crawling.
To sum up, you need to identify where you'll get your data from online (the income data & personal will be the trickiest), then identify how you want to obtain that data (manually or programmatically), and finally format the data into a usable dataset (CSV & JSON being simple and universal).

Talk to Stephen Upvote • Share
•••
Share Report

Answer URL

Share Question

  • Share on Twitter
  • Share on LinkedIn
  • Share on Facebook
  • Share on Google+
  • Share by email
About
  • How it Works
  • Success Stories
Experts
  • Become an Expert
  • Find an Expert
Answers
  • Ask a Question
  • Recent Answers
Support
  • Help
  • Terms of Service
Follow

the startups.com platform

Startups Education
Startup Planning
Access Mentors
Secure Funding
Reach Customers
Virtual Assistants

Copyright © 2025 Startups.com. All rights reserved.