Loading...
Answers
MenuScraping information of shopping websites according to price ?
For open APIs this is possible but how to do with the closed ones ?
Answers
You can either dissect the inner workings of ajax calls (if the website has any) or write a regular full-page scraper which traverses the product catalog.
Popular choices for scraping are casperjs, beautiful soup, mechanize, etc. Each having its strengths and drawbacks - casper for instance can emulate the browser behavior, while the rest are much less resource intensive but often require going through extra hoops when the website is complicated.
We have some in house technology WCML that we use to mine data on the web. Let me know if we can be of assistance.
Web Scraping has become a pretty common practice on the Internet. There are many different web scraping toolkits now and you can generally get product and price information, as long as it is part of the HTML. In general, you would not do a lot of decisioning at the scrape level but would gather all information in a database and sort from there.
The challenge is that web scraping is dependent on how the web page is structured in the HTML. As the store's pages change over time, you will need to address your scraping.
Your competitor selection should start from the closest, then you can move to a broader market. For that, you can start by creating a list of potential e-commerce platforms that can prove to be your competition. You can include as many competitors as you think are suitable. All the visible competitors are on your list. The whole process of research will help you create a huge list of competitors. Most of the time, e-commerce sites present a huge amount of data to crunch. The complicated part is to extract this data. This way, you can manage the whole price extraction without any hassle. The amount of data is going to be huge. You can go with a third-party storage system to keep your data safe and secure. This will allow you to access the extracted data from anywhere anytime. Following all the above-mentioned steps, you will conveniently gather relevant and valuable price data without spending much time.
You can read more here: https://www.promptcloud.com/blog/retailers-guide-to-price-scraping/
Besides if you do have any questions give me a call: https://clarity.fm/joy-brotonath
Related Questions
-
Which is the best hosted free bug tracking tool for a team size of up to 10 members?
Being a freelance developer for the last 10 years or so, I have seen and used almost every project management software you can think of, open-source and closed, and I have found that the "best bug tracking tool" completely depends on your process. I employ and direct teams in an Agile process that involves loose story-based requirements with point-based estimates, two-week cycles of iterative development, planning and retrospective sessions, and look-ahead and show-and-tell meetings with the stakeholders. It's important to me, then, that my tools have a method of capturing all those pieces with as much detail as I need but no more. On projects, I've successfully used Pivotal Tracker, Unfuddle, Redmine, Codebase HQ, Basecamp, Trello, and many others. I usually recommend people to Trello for light, agile management; it's essentially a digitized version of sticky notes and swim lanes. If your team actually needs a full-featured ticketing system, see Redmine (Rails), Trac (Python), or Mantis (PHP), depending on your language preference. Hosting a version of these yourself is fairly trivial, and numerous options exist for cheap or free hosted versions. For something in the middle, Github Issues is a good fit and the open-source clone GitLab.org replicates most of those features nicely. However, if you're looking for a hosted version, you're probably looking to offload that tricky "backup" thingy, and in that case, how important is your data? How proprietary? What's your business model? If you're working on an open-source project, Github will give your team a free account with private repos, issues, wikis, and the like. If you're okay with your project being "readable by all", public projects on Pivotal are still free. I'm mostly a consultant these days, leading small groups of junior or intermediate developers into a more productive, more mature, fully operational teams of senior software developers. One of the first things I teach folks is how to use a project management system... and why! It'll save your bacon if it's simple, effective, and reliable. With a few minutes of discussion about your project, I can probably help you select the right tool and service for your team. Let me know if I can help with that. Best of luck!DR
-
How can a small offshore development company find companies/software sales people to sell their service in the US/UK?
My company does a lot of consulting with offshore firms who are looking for a way to generate new business, so I hear this question a lot. My first reaction is that you need to totally reverse your mindset when you talk about your own company. You mentioned that you have: a great software developers team, proven track record, passion, real value But, everyone says that. There a 10,000 companies that have those things, so a customer isn't going to notice it. You need to figure out what your company is best at (doesn't have to be technical) and present it as a solution to a specific problem that clients have. Maybe a speciality, or really good project management, really good communications, a special expertise or experience, a personality, experience with a certain type of client.. really anything.. But, there must be some thing that makes your company 'special' otherwise you will be lost in the mix. Don't worry about things like rates, or the fact that you have 'great' developers. Those are generic. Think about why a client would really choose you, and try to build on that! After you understand your company identity, it gets much easier to identify and engage marketing channels because you understand your target.DH
-
I have a great app idea, and I need help bringing it to life.
I'm not sure if this is how you imagine this world to work, but at least according to the order you wrote it "raising funds" was first. In reality it should actually be one of your final steps of the stage you are at right now. It may even come after a year or two! So you have this great app idea, and you're looking for a place to start... Don't! Don't start yet before you decide whether you have what it takes to get into a roller coaster that can ruin your life and make you miserable! Not trying to scare you but I think most people only hear about these great success stories. They have this dream of maybe, possibly, becoming the next big thing... Because they have the best idea for an app... You don't hear about the failures so often. And even if you do, you don't hear about what the founders of these failing startups had to go through. Truth is you are most likely gonna fail. And I'm saying that without even knowing what your idea is. There are so many barriers on your way that even a great product with a great team is likely to fail. Some people would say "I'm not afraid of failing", "It's good to fail cause you learn", "Failing will make me stronger for the next startup". That's somewhat true but it doesn't mean that failing is easy. As oppose to what people sometimes say - you do not want to fail! It's very painful!!! You have to understand what failing in a startup means. You can work your a$s for 2-3 years, have little to no salary, waste other people's money (most likely your friends and family first), lose friends, fight with your partners, your family, your spouse, devote 20 hours a day for your startup all this time, forget about the little and big things you used to enjoy in life, and only then, after debating 100 times whether you should quit or not, you finally decide that it's not gonna work and you've failed. Disappointing your family, your investors, yourself. Trust me it is painful. Are you sure you wanna do this to yourself? If yes, give me a call. I have the experience you need! From idea stage, to proof of concept, to running beta tests, getting millions of millions of users in ways you can't even imagine, creating features and experience that will make these millions of users completely addicted and viral, raise money in a smart way, hire the right people, find a great co-founder, succeed, fail, be persistent, and enjoy the ride! Good luck, RoyRM
-
How do I run a closed beta test for my mobile application? Development will be finished in 3 weeks.
You should try to engage people using social networks, it is easier to spread than email. The conversion rate on emails are low but is still a valid tool for that. Send and email with a simple and objective message that will make people want to try. The best way to have feedback from users is to watch them use the app. You should put them on the hands of everybody that you can and without any instrucions and just watch, don´t even say that the app is yours. Try to do it a lot. If you want feedback from others, you can include the feedback form inside the app and suggest users to answer occasionally. I would also strongly recommend to use a tool as Flurry Analytics. Is the best way to get data from how is the use of the application. Pay attention to those data and be open to change your app a lot, you may need more features or cut some off to make it easier to use. If you need more help please contact me.BS
-
How should the dynamic between a ux designer and a developer who are working together look like?
It depends a lot of in the skill sets and experience of both people but in most cases the ux designer should be controlling the developer pretty heavily in order to make sure his ideas come through properly. The UX designer may just need to work on his approach so people don't feel bossed around and more like they are working together. In an ideal world, there would be a project manager who makes sure everything is communicated well and keeps the dynamic feeling great.JM
the startups.com platform
Copyright © 2025 Startups.com. All rights reserved.