• rumba@lemmy.zip
    link
    fedilink
    English
    arrow-up
    119
    ·
    16 hours ago

    Unless I’m misreading it which is possible it’s awfully late, he said he processed 60,000 rows didn’t find what he was looking for but his hard drive overheated on the full pass.

    Discs don’t overheat because there was load. Even if he f***** up and didn’t index the data correctly (I assume it’s a relational database since he’s talking about rows) The disc isn’t just going to overheat because the job is big. It’s going to be lack of air flow or lack of heatsink.

    I guarantee you he was running on an external NVMe, and one of those little shitty-ass Chinese enclosures. Or maybe one of those self immolating SanDisk enclosures. Hell, maybe he’s on a desktop and he slept a raw NVMe on his motherboard without a heatsink

    There are times when you want a brilliant college student on your team, But you need seasoned professionals to help them through the things they’ve never seen before and never done before.

    • iAvicenna@lemmy.world
      link
      fedilink
      arrow-up
      3
      ·
      36 minutes ago

      yes but also why say 60K when you could have literally said anything? I mean surely the fact that he thinks 60K rows a big number is already explaining alot lol.

      • Melvin_Ferd@lemmy.world
        link
        fedilink
        arrow-up
        1
        ·
        1 minute ago

        It’s bait.

        They probably an explanation tweet at the ready to make more sense of it. They just want enough 'hurr durr these idiot" comments before they reverse Uno card this with more context.

    • ameancow@lemmy.world
      link
      fedilink
      English
      arrow-up
      7
      arrow-down
      1
      ·
      edit-2
      3 hours ago

      There are times when you want a brilliant college student on your team, But you need seasoned professionals to help them through the things they’ve never seen before and never done before.

      Honestly, any sweet, white-haired old lady who keeps pictures of her dogs and grandkids on her desk who’s been doing data entry for 15 years could do circles around these clowns.

      But she might also have the wisdom and perception to know we’re not supposed to be doing this “work” at all, which is why he recruits naive teenagers and college kids who are still emotionally immature to think that this is going to be their “destiny” or their opportunity to get into the big leagues of business.

      • rumba@lemmy.zip
        link
        fedilink
        English
        arrow-up
        2
        ·
        1 hour ago

        I keep hearing things about these hires he has, I don’t think they’re naive, At least not as such. They seem to be more power hungry trust fund babies.

        But yeah, people with a few years in them would be a moral liability in that line of work.

        • easily3667@lemmus.org
          link
          fedilink
          English
          arrow-up
          1
          ·
          32 minutes ago

          Yeah if you read more of these guys tweets they are clearly in politics. One message tried to claim trump loves kids (to be clear: in the abstract sense, not in the he definitely fucked kids on an island with Epstein sense). Then they tried to twist the words to say “why don’t you love kids”. It was clumsy like you’d expect from someone who is practically a teenager, but the core is an attempt to follow the usual right wing playbook.

    • exu@feditown.com
      link
      fedilink
      English
      arrow-up
      73
      arrow-down
      1
      ·
      14 hours ago

      Can’t be a relational database, Musk said the government doesn’t use SQL.

    • Deathray5@lemmynsfw.com
      link
      fedilink
      arrow-up
      8
      ·
      12 hours ago

      Somehow I feel over clicking without understanding of the consequences sounds like something a techbro would do

  • BenLeMan@lemmy.world
    link
    fedilink
    arrow-up
    70
    ·
    17 hours ago

    You’re not supposed to place your laptop directly in the lap of your fur suit. Always leave an air gap for ventilation, smh.

  • LillyPip@lemmy.ca
    link
    fedilink
    arrow-up
    88
    ·
    17 hours ago

    This cannot be real, wtf. This is cartoon levels of ineptitude.

    Or sabotage by someone heading out? Please let this be resistance sabotage they haven’t noticed yet.

    • turnip@lemm.ee
      link
      fedilink
      English
      arrow-up
      45
      ·
      edit-2
      17 hours ago

      You guys arent running your software off raspberry pi’s with sdcards from the gas station?

      My allowance is 5$ a month!

  • nonentity@sh.itjust.works
    link
    fedilink
    arrow-up
    49
    ·
    17 hours ago

    Either she knows something novel, where processing data using voice coils is somehow beneficial, or is someone who calls their computer a ‘hard drive’, which summarily negates any legitimacy of technical competence.

  • GreenKnight23@lemmy.world
    link
    fedilink
    arrow-up
    60
    ·
    20 hours ago

    I smell something, but it’s not overheating electronics.

    I’ve processed over 5 million records on a laptop that’s almost 10 years old. it took two days to get my results.

    there’s no way 60,000 records overheated ANYTHING.

    • wewbull@feddit.uk
      link
      fedilink
      English
      arrow-up
      23
      arrow-down
      2
      ·
      19 hours ago

      Doesn’t actually say that 60k overheated his drive. He says that he ran a run on 60k, and that he couldn’t do the whole database due to overheating. Two unrelated statements except that 60k is the lower bound for what he could process.

      Doesn’t mean he knows what he’s doing though, as pretty huge datasets are processable on quite modest hardware if you do it right.

      • GreenKnight23@lemmy.world
        link
        fedilink
        arrow-up
        12
        arrow-down
        1
        ·
        16 hours ago

        that’s somehow worse.

        a “data analyst” couldn’t cut up the work into a parallel processes and run them synchronously? what the actual fuck?

        “sorry, I can only do 60k at a time.”

        just fucking split them up into 6 parallel batch processes running 10k at a time. it’s fucking math, not rocket science. I’m not even an analyst and I could fucking do that much.

  • RussianBot8453@lemmy.world
    link
    fedilink
    arrow-up
    97
    ·
    22 hours ago

    I’m a data engineer that processes 2 billion row 3000 column datasets every day, and I open shit in Excel with more than 60k rows. What the hell is this chick talking about?

    • zenpocalypse@lemm.ee
      link
      fedilink
      English
      arrow-up
      26
      ·
      20 hours ago

      Seems like a good excuse to someone who doesn’t know what they’re doing and needs an excuse because why they haven’t completed it yet?

      The whole post is complete bs in multiple ways. So weird.

      • psivchaz@reddthat.com
        link
        fedilink
        arrow-up
        3
        ·
        10 hours ago

        It sounds like Hollywood tech lingo. Like when you’re watching a movie or a TV show and the designated techy character starts just saying computer words that make no actual sense in the real world, but I guess in CSI: Idiottown the hard drives have severe overheating issues.

      • Akasazh@feddit.nl
        link
        fedilink
        arrow-up
        5
        ·
        19 hours ago

        If you work for a boss that fundamentally misunderstands what you are doing, then misleading them into thinking you’re ‘hard at work, making decisions with consequences’ is the theatre you put up to keep the cash flowing.

        It’s one of the fundamental flows of autocracy, people try and represent what you want them to

    • person420@lemmynsfw.com
      link
      fedilink
      arrow-up
      24
      ·
      21 hours ago

      Some interesting facts about excel I learned the hard way.

      1. It only supports about a million or so rows
      2. It completely screws up numbers if the column is a number and the number is over 15 digits long.

      Not really related to what you said, but I’m still sore about the bad data import that caused me days of work to clean up.

      • driving_crooner@lemmy.eco.br
        link
        fedilink
        arrow-up
        8
        ·
        edit-2
        17 hours ago

        It completely screws up numbers if the column is a number and the number is over 15 digits long.

        I work in insurance in Brazil, by standards of our regulatory body, claims numbers must be a string of 20 numbers (zfill(20) if needed). You can’t imagine the amount of times excel had fucked me up rounding down the claim numbers, this is one of the first things I teach to my interns and juniors when they’re working with the claims databases.

      • Mniot@programming.dev
        link
        fedilink
        English
        arrow-up
        9
        ·
        19 hours ago

        The row limitation seems, to me, like an actually-good thing. Excel is for data where you might conceivably scroll up and down looking at it and 1M is definitely beyond the ability of a human even to just skim looking for something different.

        An older version of Excel could only handle 64k rows and I had a client who wanted large amounts of data in Excel format. “Oh sorry, it’s a Microsoft limitation,” I was thrilled to say. “I have no choice but to give you a useful summarization of the data instead of 800k rows (each 1000 columns wide) of raw data.”

        • frezik@midwest.social
          link
          fedilink
          arrow-up
          3
          ·
          edit-2
          8 hours ago

          Some time ago, I heard a story of CS and Econ professors having lunch together. The Econ professor was excited that Excel was going to release a version that blew out the 64k row limit. The CS professor nearly choked on his lunch.

          Dependence on Excel has definitely caused bad papers to be published in the Econ space, and has had real world consequences. There was a paper years ago that stated that once a country’s debt gets above 120% of GDP, its economy goes into a death spiral. It was passed around as established fact by the sorts of politicians who justify austerity. Problem was, nobody could reproduce the results. Then an Econ undergrad asked the original author for their Excel spreadsheet, and they found a coding error in the formulas. Once corrected, the conclusion disappeared.

  • jkercher@programming.dev
    link
    fedilink
    English
    arrow-up
    36
    ·
    19 hours ago

    60k rows of anything will be pulled into the file cache and do very little work on the drive. Possibly none after the first read.

    • wise_pancake@lemmy.ca
      link
      fedilink
      arrow-up
      6
      ·
      17 hours ago

      Are you telling me there’s a difference between an inner and a cross join?

      Cross join is obviously faster, I don’t even have to write “on”

  • zalgotext@sh.itjust.works
    link
    fedilink
    arrow-up
    110
    arrow-down
    1
    ·
    1 day ago

    my hard drive overheated

    So, this means they either have a local copy on disk of whatever database they’re querying, or they’re dumping a remote db to disk at some point before/during/after their query, right?

    Either way, I have just one question - why?

    • Bosht@lemmy.world
      link
      fedilink
      English
      arrow-up
      49
      ·
      22 hours ago

      I’d much sooner assume that they’re just fucking stupid and talking out of their ass tbh.

      • kautau@lemmy.world
        link
        fedilink
        arrow-up
        20
        ·
        edit-2
        18 hours ago

        Same as Elon when he confidently told off engineers during his takeover of Twitter or gestures broadly at the Mr. Dunning Kruger himself

        Wonder if it’s an SQL DB

        Elon probably hired confident right wingers whose parents bought and paid their way through prestigious schools. If he hired anyone truly skilled and knowledgeable, they’d call him out on his bullshit. So the people gutting government programs and passing around private data like candy are just confidently incorrect

      • spooky2092@lemmy.blahaj.zone
        link
        fedilink
        English
        arrow-up
        57
        ·
        24 hours ago

        Plus, 60k is nothing. One of our customers had a database that was over 3M records before it got some maintenance. No issue with overheating lol

        • surph_ninja@lemmy.world
          link
          fedilink
          arrow-up
          30
          ·
          edit-2
          23 hours ago

          I run queries throughout the day that can return 8 million+ rows easily. Granted, it takes few minutes to run, but it has never caused a single issue with overheating even on slim pc’s.

          This makes no fucking sense. 60k rows would return in a flash even on shitty hardware. And if it taxes anything, it’s gonna be the ram or cpu- not the hard drive.

          • T156@lemmy.world
            link
            fedilink
            English
            arrow-up
            2
            ·
            edit-2
            12 hours ago

            In my experience, the only time that I’ve taxed a drive when doing a database query is either when dumping it, or with SQLite’s vacuum, which copies the whole thing.

            For a pretty simple search like OP seems to be doing, the indices should have taken care of basically all the heavy lifting.

        • AThing4String@sh.itjust.works
          link
          fedilink
          arrow-up
          16
          ·
          23 hours ago

          I literally work with ~750,000 line exports on the daily on my little Lenovo workbook. It gets a little cranky, especially if I have a few of those big ones open, but I have yet to witness my hard drive melting down over it. I’m not doing anything special, and I have the exact same business-economy tier setup 95% of our business uses. While I’m doing this, that little champion is also driving 4 large monitors because I’m actual scum like that. Still no hardware meltdowns after 3 years, but I’ll admit the cat likes how warm it gets.

          750k lines is just for the branch specific item preferences table for one of our smaller business streams, too - FORGET what our sales record tables would look like, let alone the whole database! And when we’re talking about the entirety of the social security database, which should contain at least one line each in a table somewhere for most of the hundreds of millions of people currently living in the US, PLUS any historical records for dead people??

          Your hard drive melting after 60k lines, plus the attitude that 60k lines is a lot for a major database, speaks to GLARING IT incompetence.

        • Korhaka@sopuli.xyz
          link
          fedilink
          English
          arrow-up
          2
          ·
          edit-2
          22 hours ago

          Pretty sure I run updates or inserts that count over 60k fairly often. No overheats. Select queries sometimes way higher.

      • Dave.@aussie.zone
        link
        fedilink
        arrow-up
        18
        arrow-down
        1
        ·
        24 hours ago

        You’ve got it all wrong, in traditional computer terminology the “hard drive” is the box that sits under the desk that collects cat fluff and cigarette tar.

        /s …?

      • Fuck spez@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        6
        ·
        21 hours ago

        I don’t think I’ve seen a brand new computer in the past decade that even had a mechanical hard drive at all unless it was purpose-built for storing multiple terabytes, and 60K rows wouldn’t even take multiple gigabytes.

      • baldingpudenda@lemmy.world
        link
        fedilink
        arrow-up
        5
        ·
        22 hours ago

        Reminds me of those 90s ads about hackers making your pc explode.

        Musk gonna roll up in a wheelchair, “the attempt on my life has left me ketamine addicted and all knowing and powerful.”

      • wise_pancake@lemmy.ca
        link
        fedilink
        arrow-up
        1
        ·
        18 hours ago

        I have when a misconfigured spark job I was debugging was filling hard drives with tb of error logs and killing the drives.

        That was a pretty weird edge case though, and I don’t think the drives were melting, plus this was closer to 10 years ago when SSD write lifetimes were crappy and we bought a bad batch of drives.

    • zenpocalypse@lemm.ee
      link
      fedilink
      English
      arrow-up
      19
      ·
      edit-2
      20 hours ago

      Even if it was local, a raspberry pi can handle a query that size.

      Edit - honestly, it reeks of a knowledge level that calls the entire PC a “hard drive”.

      • T156@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        arrow-down
        1
        ·
        edit-2
        12 hours ago

        Unless they actually mean the hard drive, and not the computer. I’ve definitely had a cheap enclosure overheat and drop out on me before when trying to seek the drive a bunch, although it’s more likely the enclosure’s own electronics overheating. Unless their query was rubbish, a simple database scan/search like that should be fast, and not demanding in the slightest. Doubly so if it’s dedicated, and not using some embedded thing like SQLite. A few dozen thousand queries should be basically nothing.

    • GoodEye8@lemm.ee
      link
      fedilink
      English
      arrow-up
      25
      arrow-down
      1
      ·
      22 hours ago

      My one question would be “How?”

      What the hell are you doing that your hard drives are overheating? How do you even know it’s overheating as I’m like 90% certain hard drives (except NVMe if we’re being liberal with the meaning of hard drive) don’t even have temperature sensors?

      The only conclusion I can come to is that everything he’s saying is just bullshit.

      • Auli@lemmy.ca
        link
        fedilink
        English
        arrow-up
        20
        ·
        22 hours ago

        They have temp sensors. But have never heard of a overheating drive.

          • Mniot@programming.dev
            link
            fedilink
            English
            arrow-up
            4
            ·
            19 hours ago

            Can we think of any device someone might have that would struggle with 60k? Certainly an ESP32 chip could handle it fine, so most IoT devices would work…

            • T156@lemmy.world
              link
              fedilink
              English
              arrow-up
              2
              ·
              12 hours ago

              Unless the database was designed by someone who only knows of data as that robot from Star Trek, most would be absolutely fine with 60k rows. I wouldn’t be surprised if the machine they’re using caches that much in RAM alone.

            • zenpocalypse@lemm.ee
              link
              fedilink
              English
              arrow-up
              4
              ·
              19 hours ago

              Right? There’s no part of that xeet that makes any real sense coming from a “data engineer.”

              Terrifying, really.

        • xthexder@l.sw0.com
          link
          fedilink
          arrow-up
          9
          ·
          edit-2
          20 hours ago

          Imo if they can’t max out their harddrive for at least 24 hours without it breaking, their computer was already broken. They just didn’t know it yet.

          Any reasonable SSD would just throttle if it was getting too hot, and I’ve never heard of a HDD overheating on its own, only if there’s some external heat sources, like running it in a 60°C room

          • T156@lemmy.world
            link
            fedilink
            English
            arrow-up
            1
            ·
            12 hours ago

            Hard Drives might do it if the enclosure is poorly designed (no ventilation), but I can’t imagine a situation where it would overheat like that that quickly, even in a sealed box. 30k is nothing in database terms, and if their query was that heavy, it would bottleneck on the CPU, and barely heat the drive at all.

    • Adalast@lemmy.world
      link
      fedilink
      arrow-up
      7
      arrow-down
      1
      ·
      23 hours ago

      Why? Because they feel the need to have local copies of sensitive financial information because… You know… They are computer security experts.

  • Psaldorn@lemmy.world
    link
    fedilink
    arrow-up
    195
    arrow-down
    1
    ·
    1 day ago

    From the same group that doesn’t understand joins and thinks nobody uses SQL this is hardly surprising .

    Probably got an LLM running locally and asking it to get data which is then running 10 level deep sub queries to achieve what 2 inner joins would in a fraction of the time.

    • _stranger_@lemmy.world
      link
      fedilink
      arrow-up
      81
      ·
      edit-2
      1 day ago

      You’re giving this person a lot of credit. It’s probably all in the same table and this idiot is probably doing something like a for-loop over an integer range (the length of the table) where it pulls the entire table down every iteration of the loop, dumps it to a local file, and then uses plain text search or some really bad regex’s to find the data they’re looking for.

      • morbidcactus@lemmy.ca
        link
        fedilink
        arrow-up
        37
        ·
        24 hours ago

        Considering that is nearly exactly some of the answers I’ve received during the technical part of interviews for jr data eng, you’re probably not far off.

        Shit I’ve seen solutions done up that look like that, fighting the optimiser every step (amongst other things)

      • indepndnt@lemmy.world
        link
        fedilink
        arrow-up
        14
        ·
        22 hours ago

        I think you’re still giving them too much credit with the for loop and regex and everything. I’m thinking they exported something to Excel, got 60k rows, then tried to add a lookup formula to them. Since you know, they don’t use SQL. I’ve done ridiculous things like that in Excel, and it can get so busy that it slows down your whole computer, which I can imagine someone could interpret as their “hard drive overheating”.

      • makingStuffForFun@lemmy.ml
        link
        fedilink
        arrow-up
        5
        ·
        edit-2
        23 hours ago

        I have to admit I still have some legacy code that does that.

        Then I found pandas. Life changed for the better.

        Now I have lots if old code that I’ll update, “one day”.

        However, even my old code, terrible as it is, does not overheat anything, and can process massively larger sets of data than 60,000 rows without any issue except poor efficiency.