Loading...
Answers
MenuI am building an image archive that will ultimately have millions of images you can search. I'd love the hear your thoughts on technologies to use.
My list so far:-
Storage - S3
CDN - CloudFront or CloudFlare
Database - ?
Front-End Framework - ?
Answers
I have worked on projects handling transfer of large image files across multiple vendors and being served on web for different targets (mobile, web etc.). I have also implemented cloud and DevOps solutions for high volume/scale of data and transactions.
While I could tell you a list of technologies which can be used - the problem here is more of design and architecture. So I will start with that. Also a lot of decisions are based on more information and the use case - so I will need information on certain aspects of your application.
1) For searching an image users will enter search terms. You will have to associate the search terms/tags to actual images. While there are multiple ways to do this association - what is the link between a search term and image in your case. Do you have a separate database/service which will link given search terms to given set of images?
2) Since you are talking about handling millions of images - they will need to be partitioned logically or bucketized for optimum way to store and search. Is there a natural distribution? Do your users access all the images all over the world?
3) For any data storage related to images - we will need to go one detail further. While relational DB might serve a lot of needs, there might be use cases where considering other storages will be needed. Some questions which will lead to answers are - what kind of data per image will be stored? How frequently this data might change? How many new additions will happen on this archive? How fast would you want the updates to be visible to users/consumers?
4) What kind of traffic do you expect roughly? This will decide a lot design around auto scaling, load balancing and other aspects.
5) I am not a front end expert - but similar principles of scaling would apply to front end stack. Though I can not necessarily answer what technologies might fit your needs best.
Related Questions
-
What is the best programming language for building multi-platform mobile software that is scalable?
I've been involved in several projects that hinged upon this question (generally start-ups or web+mobile apps), and it's not a clean or easy answer unfortunately! Plus every developer you talk to is going to try to sell you their services, but what you need is simply the truth! I'd suggest that you consider shifting your focus away from finding a best language. That sort of premise can eliminate options that are actually quite valid solutions. Keep in mind that any mobile app will require multiple languages working in harmony - all while used in a commercially consistent and standards-based manner. That's the part that determines your technical scalability! Almost every programming language can achieve this functionality with an experienced developer on your team. The best advice I can give you, is to consult with at least two people on this, and three if your app is os-specific. The most important insight will come from a "full stack" developer. One who's got a variety of enterprise experience, and can code at all levels of the stack. This person needs to have experience in leading a team of other developers, which forces the strategy of which technologies to use and why onto their plates daily. Secondly, you'll want to consult with a "front end" developer, who can tell you what's possible using advanced OOP JavaScript techniques (like Google's angular.js), because in an absurdly fast change over the past 3 years, much of the formerly back-end work has made it's way to the front-end, and is driven by JavaScript, predominantly JSON data, and awesome API's. The game has changed and the front-enders are the poor souls dealing with this rapid shift daily. To do this, they're also fantastic JavaScript programmers, which is a language that runs on all mobile devices and all browsers, too. Love it or hate it JavaScript is the most commonly used language in the world. Finally, if your app is OS-specific, you'll want to consult with a developer who works predominantly with the OS your app is built for. This person lets you know what's possible from the device POV, should know what stacks and JavaScript approaches can and can't be done on that OS, and how to leverage the resources of the device for your app as well as extend it's functionality. BONUS - loop back to the full stack developer to double-check the claims of the front-ender and the mobile developers. Always double check with your most senior programmer ;) All of these consults together in addition to your own research and due diligence will get you comfortable and allow you to navigate on this rather daunting but deeply important journey. There aren't any turn-key options. Instead it's a series of inter-connected modules driven by different languages, and all working in tandem. Every solution will have bugs, and no one group of technologies can do everything without proper developers. If you'd like to go further down the rabbit hole, then we should definitely set-up some time to speak. Otherwise, I wish you great luck in research and encourage you to learn as much as you can! :) It's going to seem hard, and might give you a headache here or there, but learn everything you can about how different technologies "talk" to each other, and then you will be able to build a map for keeping your app and business scalable regardless of the changing tides of technology!MM
-
What is the best way to do a permissions system for a low traffic administration system, PHP preferred, but general suggestion is also valuable.
Hi, I have been a PHP developer for 13 years and have experience building enterprise applications. What framework are you using? Most PHP frameworks have a built in authentication and authorization mechanism. For example, Yii has a builtin RBAC system that allow you to define a hierarchical permission system. If you're not using a framework or you just want to implement your own system for whatever reason then you can utilize Object Oriented Design and implement a Base controller (if MVC) that checks the permission for each request. That won't use a lot of if statements. Probably need more info, to give definitive guidance. Hope that helps.AP
-
Best way to fund my app for a startup? The programming phase is a lot of money.
Hi, check outmy response to another question similar to yours. I hope it helps. Overall what you need to do is grow your community of users, or depending on the app generate enough sales or interest typically before any investors considering joining you. Not impossible to get investors early on, but your best bet is through kickstarted if anything. My responses tend to get UpVoted by other experts. Check it out https://clarity.fm/a/8032HV
-
What is the viability of big data in education? Is learning analytics a potentially viable market considering current restraints on public education?
Qualifications on the answer: Created the first website to be ever commercially licensed by a ministry of education for in-school use (Brainium 1996). Was a VC associate at a 500,000,000 firm called Knowledge Universe focused on all things education-related that intersected with technology. Short Answer: Yes and no. Longer Answer: Learning analytics is one of those "lightening in a bottle" kind of industries. Everyone knows it's going to happen, whomever makes it happen at scale first, will have a huge advantage, and even more niche plays in this space will still create massive value for shareholders and significant transformation of education as we know it. That's the good news. The problem is that funding startups in the education space that are dependent on "permission" being granted by existing institutions and their employees is very much hit or miss. It's getting easier to raise a $500-800k seed round for a good idea with a good team, but the problem is that getting the next bigger round is very challenging in that the models of most of these seed-funded companies are not growing fast enough to prove to institutional investors that a growth model of the idea being very big in a relatively short time-period is credible. So if you're intent on pursuing this, I'd be sure to focus entirely on how you can get growth and adoption with requiring little to no institutional buy-in which sounds almost impossible on the surface of the area you're exploring but I believe isn't actually impossible. I'm very passionate about this space and want something like what you're describing to succeed so would be happy to do a quick call to hear where you're at and see if I can provide you some actionable ideas on how to reduce the friction / improve adoption of your analytics product.TW
-
How can I aggregate data from online sources about a specific topic?
There are so many ways to do it... Do you need this data for yourself, or you are planning to make a product around it? From what I see you can use Twitter API and Facebook Graph API (Are you comfortable programming?) Most of the students are active on social media so you will find lots of data. Facebook graph API will give you a number of likes and comments to all the posts of you competitors. You can analyze all the posts of your competitors. Using Twitter API you can get all the twits that use certain hashtags or mentions. If you are not into coding, but still want to get social media information, you can take a look at tools like IBM Watson ANalytics ($30 for personal use), it natively connects to Twitter API, and you don't have to be a programmer at all. It is intuitive and easy to learn. Analytics Canvas connects to Facebook Graph API (it's free for 30 days of trial). Unfortunately, you would not be able to collect any personal information from social media at large scale (age, income, gender, etc.), because it violates all the laws about privacy on the Internet. You can use census data instead. Google Sheets are a very handy tool if you are planning to use this information for personal research. You can set up a spreadsheet and add some Java script to make it collect all information from competitor's blogs, and also sites like Reddit. Finally, you can try web scraping (it's not the best, but can speed up the process). A tool like OutWitHub will collect information from websites (such as website reviews) based on the structure you provide (select html tags). You can collect thousands of reviews in one day if you automate it (paid version). Very easy to use. Note: not all the websites are open to this method, review their policies to make sure you are not violating their terms of service. Reviews belong to the website where they were published. If you REALLY need personal data (like how much they earn and how much they spend, etc.), just print out 100 questionnaires and go to Student Union Building of Dalhousie University. Most of the students will share any personal data in exchange for a Tim Horton's gift card that gets them a free coffee. It is probably the least technical and fastest way to get all the data you need. Hope this helps.OT
the startups.com platform
Copyright © 2025 Startups.com. All rights reserved.