Loading...
Answers
MenuI am building an image archive that will ultimately have millions of images you can search. I'd love the hear your thoughts on technologies to use.
My list so far:-
Storage - S3
CDN - CloudFront or CloudFlare
Database - ?
Front-End Framework - ?
Answers
I have worked on projects handling transfer of large image files across multiple vendors and being served on web for different targets (mobile, web etc.). I have also implemented cloud and DevOps solutions for high volume/scale of data and transactions.
While I could tell you a list of technologies which can be used - the problem here is more of design and architecture. So I will start with that. Also a lot of decisions are based on more information and the use case - so I will need information on certain aspects of your application.
1) For searching an image users will enter search terms. You will have to associate the search terms/tags to actual images. While there are multiple ways to do this association - what is the link between a search term and image in your case. Do you have a separate database/service which will link given search terms to given set of images?
2) Since you are talking about handling millions of images - they will need to be partitioned logically or bucketized for optimum way to store and search. Is there a natural distribution? Do your users access all the images all over the world?
3) For any data storage related to images - we will need to go one detail further. While relational DB might serve a lot of needs, there might be use cases where considering other storages will be needed. Some questions which will lead to answers are - what kind of data per image will be stored? How frequently this data might change? How many new additions will happen on this archive? How fast would you want the updates to be visible to users/consumers?
4) What kind of traffic do you expect roughly? This will decide a lot design around auto scaling, load balancing and other aspects.
5) I am not a front end expert - but similar principles of scaling would apply to front end stack. Though I can not necessarily answer what technologies might fit your needs best.
Related Questions
-
How do I find the Total Addressable Market for a Big Data product like Datameer?
As someone who has built TAM models for various industries - I can attest this isn;t always easy. Many analyst firms (former analyst here) typically do some surveys and research among public company revenue reports for the year and then do some pixie dust extrapolating. The hard thing about a market where startups like Datameer play is that it is a) a nascent market and b) not a "zero sum" game. This makes it difficult to a) fully understand WHICH companies actually are in the market for this flavor of BI and b) know how the space is developing in terms of white space AND disrupting the entrenched BI giants. You might want to look at the press published versions (read: free) of analyst market share reports, and extrapolate from their. Also, back-channel gossip may give you some revenue numbers on these players - and if you assume pipeline is 3-4x revenue, build a model from there. Hope this helps.MS
-
How can I know when to go with my gut or when to trust the data when making a business decision?
In my experience coaching hundreds of people, anytime you go against your gut (aka intuition) you will lose. Data is important to consider, but your gut often knows the path.RC
-
How can I assess the value of my data assets?
I have been helping clients design, build, and deploy data platforms for many years. Initially on premise, but now in the cloud. This is a really good question. My immediate thought is, yes, there is value in this data. The more difficult one to answer is where and how? The obviously answer is monetisation i.e. what would people or organisations be willing to pay for this data. More importantly, is this my data (vs. customer data) to sell? The data may also represent your intellectual property or be your "secret sauce" so you might not necessarily want to sell it. It is then worth asking questions like "what would the impact on my organisation be if I did not have this data?" or "what would the impact on my organisation be if my competitors had this data?". Hopefully this has given you some ideas. Feel free to book in a call if you have any questions or would like to delve into the detail.GR
-
What is the best programming language for building multi-platform mobile software that is scalable?
I've been involved in several projects that hinged upon this question (generally start-ups or web+mobile apps), and it's not a clean or easy answer unfortunately! Plus every developer you talk to is going to try to sell you their services, but what you need is simply the truth! I'd suggest that you consider shifting your focus away from finding a best language. That sort of premise can eliminate options that are actually quite valid solutions. Keep in mind that any mobile app will require multiple languages working in harmony - all while used in a commercially consistent and standards-based manner. That's the part that determines your technical scalability! Almost every programming language can achieve this functionality with an experienced developer on your team. The best advice I can give you, is to consult with at least two people on this, and three if your app is os-specific. The most important insight will come from a "full stack" developer. One who's got a variety of enterprise experience, and can code at all levels of the stack. This person needs to have experience in leading a team of other developers, which forces the strategy of which technologies to use and why onto their plates daily. Secondly, you'll want to consult with a "front end" developer, who can tell you what's possible using advanced OOP JavaScript techniques (like Google's angular.js), because in an absurdly fast change over the past 3 years, much of the formerly back-end work has made it's way to the front-end, and is driven by JavaScript, predominantly JSON data, and awesome API's. The game has changed and the front-enders are the poor souls dealing with this rapid shift daily. To do this, they're also fantastic JavaScript programmers, which is a language that runs on all mobile devices and all browsers, too. Love it or hate it JavaScript is the most commonly used language in the world. Finally, if your app is OS-specific, you'll want to consult with a developer who works predominantly with the OS your app is built for. This person lets you know what's possible from the device POV, should know what stacks and JavaScript approaches can and can't be done on that OS, and how to leverage the resources of the device for your app as well as extend it's functionality. BONUS - loop back to the full stack developer to double-check the claims of the front-ender and the mobile developers. Always double check with your most senior programmer ;) All of these consults together in addition to your own research and due diligence will get you comfortable and allow you to navigate on this rather daunting but deeply important journey. There aren't any turn-key options. Instead it's a series of inter-connected modules driven by different languages, and all working in tandem. Every solution will have bugs, and no one group of technologies can do everything without proper developers. If you'd like to go further down the rabbit hole, then we should definitely set-up some time to speak. Otherwise, I wish you great luck in research and encourage you to learn as much as you can! :) It's going to seem hard, and might give you a headache here or there, but learn everything you can about how different technologies "talk" to each other, and then you will be able to build a map for keeping your app and business scalable regardless of the changing tides of technology!MM
-
What is the viability of big data in education? Is learning analytics a potentially viable market considering current restraints on public education?
Qualifications on the answer: Created the first website to be ever commercially licensed by a ministry of education for in-school use (Brainium 1996). Was a VC associate at a 500,000,000 firm called Knowledge Universe focused on all things education-related that intersected with technology. Short Answer: Yes and no. Longer Answer: Learning analytics is one of those "lightening in a bottle" kind of industries. Everyone knows it's going to happen, whomever makes it happen at scale first, will have a huge advantage, and even more niche plays in this space will still create massive value for shareholders and significant transformation of education as we know it. That's the good news. The problem is that funding startups in the education space that are dependent on "permission" being granted by existing institutions and their employees is very much hit or miss. It's getting easier to raise a $500-800k seed round for a good idea with a good team, but the problem is that getting the next bigger round is very challenging in that the models of most of these seed-funded companies are not growing fast enough to prove to institutional investors that a growth model of the idea being very big in a relatively short time-period is credible. So if you're intent on pursuing this, I'd be sure to focus entirely on how you can get growth and adoption with requiring little to no institutional buy-in which sounds almost impossible on the surface of the area you're exploring but I believe isn't actually impossible. I'm very passionate about this space and want something like what you're describing to succeed so would be happy to do a quick call to hear where you're at and see if I can provide you some actionable ideas on how to reduce the friction / improve adoption of your analytics product.TW
the startups.com platform
Copyright © 2025 Startups.com. All rights reserved.